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OVERVIEW 


This  block  has  as  Inputs  all  of  the  outputs  of  Phase  I:  Tasks 
selected  for  training,  JPNs,  and  training  setting  selections.  The 
tasks,  elem-ints,  and  JPMs  are  carefully  reviewed  and  converteo  into 
learning  objectives.  Learning  objectives  describe  precisely  what  Is 
''0  be  learned  In  terns  of  the  expected  student  performance  under  speci- 
fier conditions  to  accepted  standards. 

These  learning  ojectives  are  necessary  to  bridge  the  gap  between 
performing  a task  and  learning  how  to  perform  a task.  Precisely 
stated  leamlns  objectives  define  for  the  student.  Instructor,  Instruc- 
tional designer,  and  test  writer  exactly  what  Is  required  of  them. 


*Port1wis  of  this  material  have  been  adapted  from:  Department  of  the  Air 
Force.  Handbook  for  designers  of  Instructional  systems;  Objectives  and 
tests  (Vol . Ill;  AFP  50-58).  klnshlngton,  D.C.:  Tteadquarters , U.S.  Air 
Force,  July  1973. 
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DCVELOP  OBJECTIVES 
1.0  INTROtKiCTlOW 

Job  tasks  for  which  tra1n1i;g  must  be  provided  have  been  Identified 
In  the  first  (^ase  of  the  ISO  incclet.  J(A  performance  measures  (JPFte} 
have  been  developed  to  measure  performance  on  these  tasks.  In  those 
cases  where  It  was  Impractical  to  measure  actual  Job  performance,  the 
JPHs  were  designed  to  provide  practical  high-fidelity  measurement 
alternatives.  Since  the  primary  objectives  of  training  1s  to  produce 
Individuals  who  can  perform  adequately  on  the  Job,  and  since  the  JPMs 
are  the  best  practical  measure  of  performance,  the  JPMs  are  the  basis 
for  developing  learning  objectives. 

Some  JPMs  closely  resemble  the  actual  task,  and  the  terminal  learn- 
ing objectives  (TLOs)  written  for  them  will  closely  resemble  the  actual 
task  statement.  Other  JPMs  will  only  approximate  the  actual  task,  but 
the  learned  behavior  should  be  virtually  Identical  to  the  behavior  on 
the  job. 

Clerical,  administrative,  and  some  technical  tasks  can  be  duplicated 
in  the  school  environment.  The  following  examples  contain  the  behavior 
portion  of  tasks  that  are  readily  converted  to  terminal  learning  objectives 
(TLOs)  in  school  settings. 

EXAMPLES 

Task  statements  from  weather  control  man. 

1.  Answer  telephone  Inquiries  about  the  weather. 

2.  Evaluate  pilot  reports. 

3.  Disseminate  weather  warnings. 
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Task  statenants  frcM  Olsburseaent  Accounting  Clerk. 

1.  CoBipiiter'  travel  allowances. 

2.  Itolntaln  Individual  travel  records. 

3..ftai1nta1n  files  of  travel  docuaents  and  records. 

Task  Statenents  from  recruiter. 

1.  Establish  rapport  and  credibility  with  prospect. 

2.  OatemlM  whether  the  prospect  aeets  the  Minimum 
requi reeents  for  service. 

3.  Uncover  the  prospects  needs  and  Interests. 

Task  state  Bents  fro*  Helicopter  Maintenance. 

1.  Align  and  balance  aaln  it^tor  blades  of  the  0H-56A. 

2.  Tijubleshoot  hydraulic  system. 

3.  Replenish  hydraulic  system. 


In  some  tasks,  only  portions  of  the  actual  required  behavior,  condi- 
tions, and  standards  from  the  job  world  can  be  represented  In  the  school 
setting.  The  following  example  contains  the  behavior  portion  of  some  of 
these  task  statements. 


EXAMPLES 

Splint  a fractured  leg 
Dispose  of  radioactive  waste 

It  has  been  estimated  that  only  15  percent  of  tasks  fall  Into  the 
category  of  those  which  canmt  completely  be  represented  In  the  school 
environment.  Normally,  though,  these  tasks  have  high  consequences  of 
Inadequate  performance  and  very  low  delay  tolerance.  Adequate  training 
of  these  tasks  is,  therefore,  critical  but  difficult. 

In  some  cases,  the  terminal  learning  objectives  (TLOs)  will  repre- 
sent the  behavior  accurately,  but  will  not  require  the  ultimate  level  of 
proficiency  that  is  required  on  the  job.  In  mar\y  troubleshooting. 
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(laintenance,  and  repair  tasks.  Job  conditions  require  speed  and  profi- 
ciency which  cannot  readily  be  achieved  In  the  school.  It  Is  expected 
that  these  high  levels  of  speed  and  proficiency  will  be  achieved  on  the 
job  after  initial  school  training. 

It  Is  Important  to  note  that  performance  of  a task  and  learning  to 
perform  a task  arc  distinctly  different.  Details  of  how  the  task  Is  per- 
formed have  alreaely  been  carefully  documented.  Now  the  concern  Is  for 
the  details  of  learning  to  perform  the  task.  A major  part  of  these 
details  is  clearly  stated  learning  objectives,  The:.e  objectives 
will  specify  what  the  learner  will  accomplish  as  a result  of  having 
received  the  instruction,  and  wiil  specify  to  the  instructional  designer 
the  exact  behaviors  the  instruction  Is  expected  to  produce. 

Prior  to  designing  instruction  to  train  individuals  to  perform  tasks, 
it  is  necessary  to  translate  the  tasks  into  terminal  leamlnq  objectives 
(TLOs)  to  be  attained  during  training.  This  means  that  each  task  must 
be  analyzed  to  determine  what  a person  needs  to  be  able  to  do  In  order 
to  perform  the  task  successfully.  During  this  analysis,  TLOs  are  broken 
down  into  their  component  parts  which  a» j documented  as  learning  objec- 
tives (LOs)  which  may  be  further  divided  into  learning  steps  (LSs).  These 
learning  objectives  will  identify  the  mental  skills.  Information,  atti- 
tudes, or  physical  skills  that  are  required  to  perform  the  TLOs. 

Terminal  learning  objectives  (TLOs)  are  the  highest  classification  of 
learning  objective  in  the  ISO  model.  TLOs  are  t?e  direct  translation  of 
the  JPMs  into  the  training  world.  Analysis  of  the  TLOs  result  in  additional 
learning  objectives  (LOs).  LOs  provide  the  level  of  detail  necessary  to 
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describe  the  major  cotponents  that  must  be  learned  In  orcter  to  perform, 
the  TLOs.  Learning  steps  (LSs)  are  <terived  through  analyzing  the  learn- 
ing objectives  (LOs)  Into  levels  of  detail  necessary  to  describe  all  the 
components  that  most  be  leaned  1n  ortter  to  perform  the  LOs. 

TLO  and  LO  statements  should  specify  the  action,  cwidltlons.  and 
standartb  which  can  be  measured  by  test  (oral,  written,  or  performance). 
TLOs  and  LOs  can  be  effectively  developed  to  support  all  four  types  of 
learning  cate^ries;  mental  skills.  Information,  physical  skills,  and 
attitudes.  TLOs  and  LOs  have  three  general  word  groups,  or  character- 
istics: actions,  conditions,  and  standards. 

EXAMPLE 

"W»en  the  trainee  completes  this  Instruction,  he  will 
be  able  to: 

(action)  Add  a colwm  of  figures  containing  10 
A-dIgIt  nunbers 

(cor.ditlon)  given  paper  and  pencil 
(standard)  in  2 minutes  correctly." 

LSs  do  not  need  to  specify  the  conditions  and  standards  but  do  need  a clear 
statement  of  the  action:  e.g.,  locate  the  clutch. 

The  procedures  for  developing  learning  objectives  are  shown  In 
Figure  II. 1,  the  foldout  page  at  the  end  of  this  block. 
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£.0  PWK:EttiR£S 

2,1  Prepare  Teralnal  Learning  Ofajcctlve  for  Each  Task 
IN  first  step  In  ^veloplng 
learning  objectives  Is  to  p.'^pare 
terBlf»1  lea-  <1ng  objectiras  (TLOs) 
which  are  direct  translations  of  the 
JP)te  Into  objectives  for  the  train- 
ing wor’J.  These  TLOs  will  Inscribe 
the  actions,  conditions,  and  stan- 
dards which  win  be  net  in  the  training 
world  to  prepare  the  persoi'i  for  the  Job  world.  ^ 

A TLO  is: 

1.  a specific  description  of  the  action  the  learner  Is  to 
exhibit  after  training, 

2.  the  conditions  under  vdilch  the  action  will  take  place,  and 

3.  the  standards  or  criterion  which  icust  be  reached. 

TLOs  are  Intencted  to  be  directly  related  to  job  task  performance  even 
when  the  action,  conditions,  and  standards  cannot  be  1dei  -1cal.  In 
many  tasks,  these  will  be  Identical  and  there  will  be  no  need  to  sacri- 
fice fidelity. 

The  TLOs  should  be  doc»®iented  on  a form  similar  to  the  one  shown 
In  Figure  II .2. 

Traditionally,  words  used  to  describe  course  content  and  objectives 
have  been  vague,  and  susceptible  to  misinterpretation.  The  learning 
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objective  provides  a more  precise  action  statement  which  presents  a 
word  picture  of  what  the  Individual  "must  be  able  to  do."  This  approach 
avoids  vague  terms  and  stresses  the  actions  the  Indlvlduil  must  perform 
to  be  satisfactory  In  a field  environment. 

EXAMPLES 

1.  Go  to  a two-hour  class  on  the  AN/PRC  77. 

2.  Use  the  AN/PRC  77  to  establish  cotimunl cations 
with  a specified  station. 

Statement  1 above  is  specific  In  telling  a person  what  to  do,  but  not 
what  is  expected.  Statement  2 tells  exactly  what  is  expected. 

Terminal  learning  objectives  (TLOs)  and  learning  objectives  (LOs) 
should  have  all  the  characteristics  listed  below.  They  will  be  evaluated 
by  these  same  criteria: 

1.  Objectives  must  be  a statement  of  student  behavior  (action), 
such  as  the  creation  of  a product  or  some  other  overt  act, 
which  can  be  accepted  as  evidence  that  the  intended  outcome 
lias  occurred. 

2.  The  behavior  must  describe  specifically  all  outcomes  that 
will  demonstrate  that  learning  has  occurred. 

3.  The  student  behavior  called  for  must  be  capable  of  observa- 
tion and  evaluation  within  the  learning  and  testing  environ- 
ments’. 

4.  The  objective  must  be  stated  in  learner  rather  than  teacher 
terms,  i.e.,  actions  which  the  student  will  perform  rather 
than  what  the  teacher  will  say  or  do. 
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5.  There  must  be  a standard  against  which  the  student  behavior 
win  be  measured.  It  must  be  fully  specified. 

6.  The  statement  of  the  conditions  under  which  the  student 
behavior  will  occur  must  be  fully  specified. 


Table  II. 1 shows  the  three  parts  of  a well-stated  learning  objec- 
tive, and  Table  1 1. 2 shows  hov.’  to  write  the  three  parts  of  the  objective. 
The  balance  of  this  section  gives  further  details  on  writing  these  three 
parts . 

2.1.1  The  .Action  Part  of  a TLO  or  LQ 

An  instructor  cannot  read  the  mind  of  the  student  to  see  how  well 
he  understands.  Only  through  some  overt  activity  of  the  student  can 
the  extent  of  his  knowledge  or  skill  be  measured.  So,  the  action  part 
of  a learning  objective  should  specifically  state  what  the  student  does. 

"Develop  an  understanding  of  Ohm's  Law"  Is  an  unspecific,  obscure 
learning  objective.  It  can  be  Interpreted  too  many  ways.  One  person 
may  feel  that  the  student's  reciting  the  law  indicates  understanding. 
Another  may  say  that  the  student  should  be  able  to  explain  the  law.  A 
third  may  contend  that  the  only  way  the  student  can  satisfy  this  learning 
objective  is  for  him  to  actually  use  the  formula  in  solving  electrical- 
circuit  problems. 

EXAHPLE 


"Given  the  mathematical  formula  for  Ohm's  Law, 
when  current  and  resistance  are  known  and  a 
slide  rule  (condition),  determine  applied  voltage 
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TABLE  II.l  The  Three  Parts  of  a Wei Instated  Learning  Objective 


CONOITION(S)  (INPUT) 

PERFORMANCE/ 
BEHAVI3R(S)  (ACTION) 

STANOARD(S)  (OUTPUT) 

Inscription  of  the 
condltlon(s)  of 
perforoance— what 
Is  presented  to 
the  student 

Description  of  the 
action  or  behavior— 
What  the  student 
Is  expected  to  do 

A statenent  of  the 
output  or  outcome 
of  the  performance 
and  the  standard(s) 
of  performance 

TABLE  II. 2 How  to  Write  the  Three  Parts  of  an  Objective 


TO  WRITE  CONDITIONS, 
SPECIFY  WHAT  STWENT 
IS  GIVEN 

TO  WRITE  PERFORHUNCE/ 
BEHAVIORS,  SPECIFY 
WHAT  STUDENT  DOES 

TO  WRITE  STANDARDS, 
SPECIFY  OUTPUT  (OR 
OUTCOME)  AND  HOW 
WELL  IT  IS  DONE 

INPUT(S)  Include: 

Use  ACTION  verbs  that 

OUTPUTS  AND  STANDARDS 

Job  Aids 

are: 

Observable 

Criteria  for  Standards; 

Equlpamnt 

Measurable 

Coa|)1eteness 

Technical 

Verifiable 

Accuracy 

References 
Special  Tools 

Reliable  (not 
prone  to  varying 

Types  of  Standards; 

Environmental 

Interpretation) 

Standard  Operating 

Conditions 

Procedure  (SOP) 

Sped  a1 

Ho  Error 

Instructions 

Signals, 

Symbols 

Time  Requi reaients 
Amount  of  Super- 

Problem 
Sltaatlons  or 

vision 

Qualitative 

Conti  ngendes 

Indices 
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This  1s  a more  precise  learning  objective.  An  objective  stated  this 
lets  everyono— the  student.  Instructor,  supervisor,  measur^nt 
personnel,  writers,  and  course  managers--know  exactly  what  the  student 
Riust  learn.  It  states  what  action  the  student  should  exhibit. 

The  action  described  In  the  learning  objective  must  allow  the  test 
writer  to  prepare  proper  Items  to  measure  It.  Clearly,  learning  objec- 
tives that  call  for  one  type  of  behavior  and  tests  that  require  another 
type  of  behavior  can  only  lead  to  confusion  and  have  no  place  In  the  ISD 
process.  If  you  find  that  the  objectives  you  have  written  are  not  con- 
sistent with  the  tests,  you  will  have  to  revise  the  test  Itens,  the 
learning  objectives,  or  both  until  they  match. 

The  main  Intent  of  the  action  verb  is  to  reflect  the  type  of  learning. 
If  we  desire  the  student  to  learn  information,  this  implies  that  he  will 
be  required  to  STATE  something.  For  mental  skills,  the  student  will  be 
required  to  DEMONSTRATE  his  ability  to  use  that  skill.  Evidence  of  the 
capability  to  perform  a physical  skill  requires  that  the  student  EXECUTE 
the  skill.  And,  attitudes  require  that  the  student  CHOOSE  among  alterna- 
tive courses  of  action. 

Using  action  verbs  reduces  ambiguity.  Unless  you  specify  student 
actions  with  an  action  verb,  the  student  will  probably  end  up  doing  some- 
thing else.  The  verbs  you  select  must  reflect  behaviors  that  are  observ- 
able, measurable,  verifiable,  and  reliable.  Use  action  verbs  that  meet 
criteria,  and  that  are  appropriate  to  the  training  and  the  test.  Some 
examples  of  suitable  veits  are  given  In  Table  11.3. 
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TABLE  1 1. 3 Examples  of  Suitable  Verbs  for 
Each  of  the  Four  Types  of  Objectives 


INFOmATION 

fttNTAL  SKILL 

PHYSICAL  SKILL 

AHITUOE 

State 

Demonstrate 

Execute 

Choose 

Name 

Discriminate 

(^rate 

Volunteer 

Recite 

Classify 

Repal r 

Allow 

Describe 

Generate  (a  solution) 

Adjust 

Recu^nd 

List 

Apply  (a  rule) 

Manipulate 

Defenu 

Relate 

Sul  ve 

Handle 

Endorse 

Tell 

Derive 

Manufacture 

Cooperate 

Write 

Prove 

Calibrate 

Accept 

Express 

Analyze 

Remove 

Decide  to 

Recount 

Evaluate 

Repi ace 

Agree 

When  the  expected  performance  has  been  successfully  analyzed  Into  the 
oroper  types  of  learning,  you  can  Indicate  clearly  what  1t  Is  that  the 
learner  Is  to  do.  These  Indications  require  precise  statements.  Each 
action  statement  can  contain  two  verb;..  The  first  may  Indicate  the  kind 
of  objective  that  is  being  written  (see  Table  11.3}  and  the  second  will 
Indicate  how  the  learner  will  perform  the  action  Indicated.  The  excep- 
tion occurs  when  there  can  be  no  confusion:  "Sing  the  Star  Sfi^angled 
Banner."  But,  "Desc''1be  the  relationship  between..."  could  easily  mean 
either  orally  or  In  writing.  So,  if  there  can  be  any  doii>t  about  the 
intention,  use  a second  verb  to  indicate  what  the  learner  Is  to  do. 

Correctly  written  action  statements  will  make  It  easier  to  add  the 
conditions  and  standards  and  will  make  it  easier  for  the  person  using 
your  learning  objectives  to  develop  Instruction. 
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2.1.2  The  Conditions  Part  of  a TLO  or  LO 


A properly  prepared  learning  objective  clearly  states  the  limits  and/ 
or  cwiditions  of  student  performance.  This  portion  of  the  learning  objec- 
tive describes  the  Important  aspects  of  the  performance  environment.  What 
does  the  student  have  to  work  witti?  Can  he  select  his  own  tools?  Are 
technical  orders  or  checklists  available  as  a starting  point?  The  con- 
ditions portion  of  the  learning  objective  should  specify  the  objects, 
events,  words,  or  symbols  which  will  be  presented  to  the  student.  Several 
sample  statements  of  conditions  are  shown  in  Table  II. 4. 


TABLE  II. 4 Sample  Statement  of  Training  Conditions 


EXAMPLE 


State  the  conditions  iii|K>sed 
upon  the  student. 


Oo  not  overuse  the  word 
"given."  Use  variations, 


Establish  the  working 
conditions . 


the  appropriate  formula,  compute  the 
surface  area  of  the  sphere. 

The  student  does  not  have  to  memo- 
rize or  derive  the  formula.  He 
solves  the  problem  using  the 
formula  provided. 

"Field-strip  and  assemble  an  M-16A1 
rifle  under  conditions  of  total 
darkness ." 

This  implies  that  the  student  will 
be  provided  with  the  rifle,  and 
requires  that  he  strip  and  assemble 
the  weapon  in  complete  darkness . 

"Usinq  available  library  resources 


prepare  a staff  stu<iy  comparing 
the  conmunl cations  support  con- 
tributions made  by  five  military 
leaders  during  the  nineteenth 
century." 
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A TLO  or  LO  require  more  than  one  statement  of  condition  In 
order  to  secure  the  desired  action.  On  the  other  hand.  In  some  cases 
the  condition  V Implied,  as  In  the  case  of  "Recite  the  Gettysburg 
Address,"  the  condition  "from  memory"  n^y  be  omitted  since  It  Is 
implied  by  the  verb  "recite." 

2.1.3  The  Standards  Pert  of  the  TLO  or  LO 

A third  requirement  for  a welt-prepared  .earning  objective  1s  a 
clearly  stated  measurable  standard  of  performance.  The  criteria  for 
standards,  completeness^  and  accuracy  are  summarized  In  Table  II. 5. 


TABLE  II. 5 Criteria  for  Standards  In  Objectives 


Criteria  foii 

GOOD  STANDARDS 


Coaqileteness 


WHAT  IS  SPECIFIED 


The  precise  nature  of  the  output. 

Number  of  features  that  output  must  contain. 

Number  of  steps,  points,  pieces,  etc.,  that  must 
be  covered  or  produced. 

Any  quantitative  statement  that  Indicates 
acceptable  portion  of  total . 


Accuracy 


How  closa  to  correct  the  performance  must  be. 

Exact  numbers  reflecting  tolerances. 

Values  or  dimensions  that  acceptable  answers/ 
performance  can  assure.  (These  may  be  qualitative.) 


1 


Time 


How  mar./  days,  hour^,  minutes,  or  seconds 
can  be  used. 
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The  student's  action  will  nesult  In  an  output,  the  quantity  or 
qMlIty  of  which  Is  the  standard  of  perfonaance.  You  nust  (teterwlne 
which  one  or  more  of  the  six  types  of  standante  shown  In  Table  II  6 
the  student  will  have  to  meet. 

TABLE  11.6  Six  Types  of  Standards  for  Objectives 


Referring  to  a 
Standard  Operating 
Procedure 


EXAH>LE 


ven  the  situational  requlrenents  for 
an  Instrument  approach,  and  the  local 
airfield  regulations,  perform  the  pro- 
ce<hire  the  pilot  follows  to  cooq>1ete 
an  Instrumnt  landing.  The  performed 
steps  win  be  In  correct  or^r  and 
will  comply  with  local  requlationT.*' 


Implying  the  standard 
of  NO  ERROR 


Specifying  mlnlmiM 
acceptable  level  of 
performance 


Specifying  the  time 
requi  resets 


Specifying  the  rate 
of  production 


Specifying  qualitative 
requi  remits 


"Given  the  diameter  of  a s. 
ai^ropriate  formula,  use  a calculator 
to  co^ute  to  two  decimal  places  the 
surface  area  of  the  sphere."  Adding 
"He  win  perform  without  error"  would 


not  Increase  the  requirement  for 
accuracy. 
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When  writing  standards,  quantify  where  possible.  (Remember, 
however,  that  the  sixth  type  of  standard  In  Table  II. 6 Is  qualitative, 
and  usually  can  be  quantified  only  through  rating  procedures.) 

1.  Many  standards  are  quantifiable: 

a.  Exact  number  of  words  typed  per  minute 

b.  Following  the  10  steps  of  a procedure 

c.  Listing  the  206  bones  In  the  human  body 

2.  Some  standards  are  not  directly  quantifiable: 

a.  adjusting  a carburetor  until  the  engine  runs  at  its 
smoothest  point 

b.  making  a patient  feel  comfortable  (you  could  follow 
a procedu>"e,  but  not  succeed  If  you  were  hostile) 

2.2  Determine  Appropriate  Learning  Category  For  Each  TLO 
Before  a learning  analysis  can  be 
made  of  the  TLOs,  the  appropriate 
learning  category  must  be  determined 
for  each  TLO.  This  is  because  each 
learning  category  Is  analyzed 
differently. 

The  ISO  model  includes  four  categories 
of  learning  in  which  most,  if  not  all,  TLOs, 

LOs,  and  LSs  fall.  These  categories  include 
the  learning  of  mental  skills,  information,  physical  skills,  and  atti- 
tudes. A description  of  each  of  these  follows: 

1.  Mental  skills;  Those  processes  of  identifying,  classifying, 

using  rules,  and  solving  problems  that  involve  active  mental 
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processing.  Mental  skills  Iniply  the  capability  of  applying 
the  learning  to  some  situation  and  demonstrating  the  n^ntal 
skill.  Mental  skills  involve  thinking,  creating,  and 
analyzing. 

2.  Informatloi  (Recalling  bodies  of  knowledge);  This  Involves 
the  learning  of  names,  labels,  facts,  nomenclature,  or 
larger  bodies  of  knowledge  that  are  necessary  for  successful 
job  performance.  Information  Invlles  that  the  person  can 
store  and  recall  the  knowledge. 

3.  Physical  skills:  These  Involve  some  physical  or  manipula- 
tive activity.  They  require  movement  of  some  of  the  muscles 
of  the  body  and  are  directly  observable  (sometimes  called 
psychomotor  skills). 

4.  Attitudes:  These  are  not  always  directly  observable  but  are 
reflected  In  the  choices  a person  makes.  Attitudes  are  indi- 
cated by  a person's  behavior.  Thus,  a person  with  a positive 
attitude  towards  job  safety  Is  one  who  chooses  to  practice 
good  safety  habits. 

To  determine  the  propeh  category  for  a particular  TLO  requires  con- 
sideration of  what  nwst  be  learned  in  order  to  master  the  TLO.  The  learn- 
ing analysis  will  then  proceed  according  to  the  guidelines  for  the  nx)st 
appropriate  category.  If  this  analysis  reveals  LOs  that  more  appro- 
priately fit  a different  category  of  learning,  you  will  further  analyze 
such  LOs  according  to  the  guidelines  for  the  new  category. 


I 


1 


I 
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2.3  Perform  Learning  Analysis  for  Each  Terminal  Learning  Ob.lecttve 
First,  analyze  each  TLO  to  determine 
the  learning  objectives  (LOs)  and 
learning  steps  (LSs)  necessary  to  master 
the  TLO.  These  LOs  and  lSs  will  give  the 
Instructional  designer  the  lev  . of  detail 
needed  to  design  Instruction.  At  this 
point,  state  clearly  all  the  assumptions 
about  the  enti^  behaviors  of  students 
for  whom  the  Instruction  will  be 
developed;  that  Is,  decide  what  skills  and  knowledges  the  typical 
student  already  has.  Later  these  assumptions  will  be  tested  and  correc- 
tions made  where  the  assumptions  were  wrong.  The  reason  for  making  these 
assumptions  here  Is  to  save  considerable  work  1n  analyzing  objectives. 

The  TLQs  need  to  be  analyzed;  that  is,  broken  down  Into  finer  details 
of  what  must  be  learned  to  the  point  that  further  analysis  Is  not 
required  since  the  students  already  can  master  the  details. 

EXAMPLE 

If  the  action  portion  of  an  LO  1s  "calculate 
mean  scores"  and  you  have  analyzed  this  to 
the  level  of  detail  of  "add  column  of  numbers," 
you  need  not  analyze  this  LO  further  If  you 
are  reasonably  certain  the  students  already 
can  add  columns  of  numbers. 

Details  of  performing  learning  analyses  for  the  four  categories  of 
learning  are  given  in  the  final  four  sections  of  this  block.  All  of 
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the  new  LOs  and  LSs  should  be  documented  on  a form  similar  to  the 
learning  objective  analysis  worksheet  shown  In  Figure  II. Z.  page  7. 

The  li^jortant  consideration  Is  to  Identify  aJX  of  the  Important 
mental  skills,  Information,  physical  skills,  and  attitudes.  Once  you 
have  completed  this  analysis,  try  to  eliminate  any  rental  skill,  1nfor 
matlon,  physical  skill,  or  attitude  that  is  not  essential  even  if  1t 
may  be  nice  to  know. 


2.3.1  Perform  Learning  Analysis  for  the  Mental  Skills  Category 

For  those  items  in  the  mental  skills  category, 
identify  those  mental  skills  that  must  be  learned 
before  a person  will  be  able  to  learn  the  rental 
skills  that  have  been  Identified.  For  example, 
if  the  rental  skill  is  that  of  decision  making, 
the  analysis  should  reveal  the  rules  and  concepts 
that  a student  must  possess  before  he  Is  able  to 
solve  the  problem;  The  purpose  of  the  analysis 
of  rental  skills  is  to  Idertiiy  those  capabilities 
tiiat  a student  must  possess  before  he  can  learn  the  skill  to  be  taught. 

Mental  skills  include  rule  using  and  learning.  Identifying  synrtjols, 
classifying,  using  verbal  infonnation,  detecting  and  making  decisions. 


EXAMPLES 


Learning  Sub-Categories  Examples  of  Learning  Objective 

Action  Staterent 


I.  Mental  Skills 

1.  Rule  learning  and  using  Select  the  proper  fire  extinguisher 

for  the  following  classes  of  fires: 
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2.  Class1fy1ng--recognl2it 9 
patterns 

3.  Identifyirg  symbols 

4.  Detecting 

5.  Making  Decisions 


a.  ordinary  contustible  materials 

b.  flatiiTiable  liquids 

c.  electrical  fires 

Identify  U.S.  and  U.S.S.R. 
destroyers 

Identify  Medical  Service  Syttt)Ols 

Visually  detect  camouflaged 
armored  vehicles  in  a terrain 
setting 

Select  a diagnostic  strategy  for 
a malfunction  in  the  main  drive 
shaft 


To  analyze  ment 

To  analyze  mental  skills,  repeatedly  ask  the  question  "What  does  the 
learner  have  to  be  able  to  do  in  order  to  do  this  task?"  The  breaking 


down  of  the  skill  into  its  corr^jonent  parts  results  in  a set  of  subskills. 
The  subskills  will  form  a hierarchy  which  will  be  helpful  later  in 
sequencing  objectives.  A learning  hierarchy  shows  the  progression  from 
simple  skills  to  more  complex.  Often,  the  TLO  will  have  many  LOs  and  LSs. 


EXAMPLE 


Start  with  the  JPM,  "Get  from  point  a to  point  b," 
then  ask  the  question  "What  does  the  learner  have 
to  be  able  to  do  to  get  from  point  A to  point  B?" 


*"lHke  with  pack"  looks 
more  like  a physical  skill, 
and  should  be  analyzed  as 
such.  Here  we  are  con- 
cerned only  with  mental 
skills. 
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Again  ask  the  same  question  for  each  subsklll: 


Avai'nb 


fo 

» W 


Copy 
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This  analysis  continues  until  the  newly  Identified  elements  are  ones 
that  the  intended  trainees  already  can  dc. 


To  Identify  the  Teaming  subskllls  for  mental  skills  ask: 

What  thought  processes  (discriminating,  identifying, 
classifying,  rule  using,  decision  making)  pMSt  a 
person  be  able  to  perform  In  order  to  achieve  this 
learning  objective? 

Although  rough  copies  of  hierarchical  charts  are  needed  1n  actually 
performing  the  learning  analysis.  In  documenting  the  results  of  the 
analysis  on  mental  skills  which  results  In  a hierarchical  structure.  It 
Is  best  to  represent  the  hierarchy  as  In  the  following  example. 


EXAMPLE 


Each  TLO  is  assigned  a letter  A-Z.  The  LOs  and  LSs  which 
make  up  that  TLO  are  specified  by  prefacing  each  LO  and 
IS  with  the  letter  of  the  TLO  followed  by  the  appropriate 
nuittering. 


GET  FROM  POINT  A TQ  POINT  E 


A.l.ll 


INTERPRET 

r^Ap 


ORIENT  MAP 
ANO  COMPASS 


HIKE  WITH  P< 


A.l.  1.1 


use  GRID 
SYSTEM 


(READ  COMPASS* 


A. 1,1.2  A.l. 2.1 


A. 1.2.2 


apply  RULES 

FOh  USING 

compass 

A.  1.2. 3 
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DOCUMENTATION  OF  ANALYSIS 

A.  Get  from  point  A to  point  B 

A.l  Orient  mao  and  compass 

A. 1.1  Interpret  map 
A. 1.1.1  Use  legend 
A. 1.1. 2 Use  system 

A.  1.2  Use  con^ss 

A. 1.2.1  Read  co^^)ass 

A. 1.2. 2 Sight  compass 

A. 1.2. 3 Apply  rules  for  using  compass 

A. 2 Hike  with  pack 

The  letter/number  combination  from  the  above  example  could  be 
Interpreted  as  the  objective  number  on  the  learning  objective  analysis 
worksheet  shown  In  Figure  II. 2. 

2.3.2  Perform  Learning  Analysis  for  the  Information  Category 


..Ulilinlll  111 
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A 


I 


EXAMPLE 


Learning  Subcategory  Exanple  of  Learning  Objective 

Action  Statewer.t 


i 

Describe  the  rules  of  war  , 

Including  forbidden  targets, 
tactics,  and  techniques; 
treatment  of  captives, 

^talnees,  civilians  and 
private  property;  and  what 
to  do  when  crimes  are 
conmltted. 

1 

i 

1 

The  learning  of  Information  (toes  not  requlir  the  stixtont  to  produce  i 

J 

or  create  arything  new.  He  may  receive  Information  and  reproduce  It 
In  his  own  words.  Isolated  pieces  of  Information  are  hard  to  remenber 
or  use.  but  some  Information  Is  learned  the  first  time  It  Is  heard  or 
read,  such  as:  Pay  day  is  every  other  Thursday. 

To  Identify  the  learning  objectives  for  Infornatlon,  ask: 

"What  Information  must  a person  know  (be  able  to  recall 
or  state)  in  order  to  achieve  this  learning  objective?" 

The  analysis  continues  until  all  relevant  facts  have  been  Identified,  and 
Include  all  Information  the  students  do  not  already  know. 

Given  a performance  oriented  task  analysis,  Informatton  tasks  should 
be  rare.  However,  for  some  contat  tasks  the  only  practical  alternative 
mcy  be  to  develop  a low  physical  fidelity  TLO  that  Is  principally  Infor- 
mation. The  reasons  may  be: 


II.  Information 
6.  Recalling  bodies 
of  knowledge 


1.  Evidence  that  “knowing"  the  Information  has  high  predictive 
validity. 
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2.  No  job  crtterlon  therefore  no  rationale  for  going  to 
simulation. 

3.  A shortage- of  training  resources  or  time. 


EXAWLES 

1.  TLO:  Describe  the  rules  of  war. 

Include: 

(1}  Forbidden  Targets,  tactics,  and  techniques: 

a. 

a.  Do  not  attack  non-cont>atants . 

b.  Do  not  cause  destruction  beyond  the  requirements 
of  your  mission. 

c.  Do  not  attack  protected  property. 

d.  Do  not  shoot  at  the  Red  Cross. 

e.  Do  not  hide  behind  the  medical  service  synt>ols. 

f.  Do  not  shoot  at  a parachute. 

g.  Do  not  use  poison  or  personal  weapons. 

h.  Do  not  alter  your  weapons  or  ammunition  to  Increase 
eneniy  suffering. 

(2)  Treatment  of  captives  and  detainees: 


(3)  Treatment  of  civilians  and  private  properties: 


(4)  What  to  do  when  crimes  are  committed: 


Conditions:  On  paper,  with  no  references. 

Standards:  All  four  areas  must  be  Included.  Within  each 
area  no  more  than  one  rule  can  be  omitted. 

2.  If  an  information  goal  is  stated  "Be  familiar  with  the  basic 
regulations  governing  the  enlistment  process"  the  action  has 
to  be  redefined  with  action  verbs.  Three  action  portions  of 
TLOs  were  derived  from  the  goal  statement. 

(1)  Determine  basic  enlistment  criteria 

(2)  Determine  enlistment  criteria  for  specific  options 

(3)  Determine  waiverable/non-waiverable  enlistment  disqualification. 
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When  you  ask  tlie  question  what  Information  must  a person  be  able 
to  recall  In  order  to  achieve  this  for  (1),  some  of  the 
LOs  are: 


a.  Given  a copy  of  the  regulation  and  a series  of  prospects, 
present  the  enlistment  options,  and 

b.  Match  the  prospects  qualifications  with  the  specific 
enlistment  criteria  for  the  prospects'  chosen  optlon(s). 


2.3.3  Perform  Learning  Analysis  for  the  Physical  Skills  Category 
In  the  analysis  of  physical  skills. 

It  is  necessary  to  specify  the  individual 
movements  and  sequencing  of  those  move- 
ments that  form  the  total  skill.  There 
are  four  subcategories  of  physical  skills: 


EXAMPLE 


Learning  Subcategories  Examples  of  Learning  Objective 

Action  Statement  

TTE  Physical  Skills 

7,  Performing  gross  Load  the  Chaparral 

motor  skills 

8.  Steering  and  Assemble  and  disassenble  the  M203 

guiding--coiit1nuous  grenade  launcher. 

movement 


9.  Positioning  movement  Conmunlcate  with  divers  at  a depth 
and  recalling  pro-  of  30  meters  using  telephone 
cedures  equipment. 

10.  Voice  cornnuni eating 


To  identify  the  learning  objectives  for  physical  skills  ask: 
What  motor  activities  roust  a person  bt  able  to  perform 
in  order  to  achieve  this  learning  objective. 
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This  analysis  continues  until  skills  are  broken  down  to  the  point  that 
the  components  are  ones  the  Intended  students  can  a1rea4y  do. 


EXAWLES 


1.  Install  Components 

In  an  AH/PRC  77 

Install  battery  Install  ant^na  Install  handset 

/ 

/ 

Although  these  require  Information  and  mental  skills  to  do  them 
correctly,  the  process  Is  also. a physical  skill  and  It  can  be 
analyzed  that  way.  The  easiest  way  to  decide  the  steps  for 
Installing  the  battery  Is  tp^wrlte  them  In  the  order  they  are 
generally  performed  on  the  job. 

1.1  Skills  Required  to  Install  Battery 

1.1.1  Turn  the  function  switch  to  the  off  position. 

1.1.2  Stand  the  Receiver/Transmitter  on  a level  surface 
with  the  front  panel  facing  downward. 

1.1.3  Simultaneously  release  the  two  battery  box  retaining 
clangs  by  pushing  the  topmost  part  of  each  clamp  down 
and  away  from  the  Receiver/Transmitter  case. 

1.1.4  Remove  the  Battery  Box  by  lifting  upward. 

1.1.5  Lay  the  Battery  Box  to  one  side  on  the  working  surface. 

1.1.6  Position  the  Battery  Connector  In  line  with  the  Radio 
Connector  until  the  connectors  are  mated. 

1.1.7  Install  the  Battery  Box  on  the  Receiver/Transmitter. 

1.1.8  Secure  the  Battery  Case  Retaining  Clamps. 

2.  In  preparation  of  AN/PRC  77  for  operation,  one  part  calls  for 
inspecting  the  battery  and  battery  box.  "Inspect  the  battery 
and  battery  box  In  3 minutes  given  an  AN/PRC  77"  could  be  a 
learning  objective.  If  the  learning  objective  Is  further 
analyzed  there  are  many  steps  Involved  In  Inspecting.  These 
could  each  be  another  LO  or  an  LS.  Since  there  are  no  con- 
ditions that  are  different  from  the  LO,  and  there  Is  no  need 
for  standards  for  the  steps,  they  can  all  be  subsumed  under 
the  LO  as  Learning  Steps. 

LO:  Given  an  AN/PRC  77  Inspect  the  battery  and  battery  box 
In  3 minutes. 

LSs: 

2.1  turns  the  function  switch  to  the  OFF  pcslt^nn 

2.2  stands  the  recelver/transniltter  (with  all  accessories 
removed)  on  a level  surface  with  the  fr^nt  panel  facing 
downward. 
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2.3  siimjltaneously  releases  the  two  battery  box  retaining 
clangs  by  pushing  the  topinost  part  of  each  clanp  down 
and  away  from  the  rcceiver/transmi tter  case. 

2.4  removes  the  battery  box  by  lifting  upward, 

2.5  Inspects  the  gasket  or  battery  box  for  damage. 

2.6  Inspects  the  battery  box  for  corrosion. 


2.17  Stand  the  receiver/transmitter  on  a level  surfree  with 
the  front  panel  facing  upward. 


I Obviously,  some  of  these  steps  In  the  above  examples  could  be  analyzed 

further;  however,  these  lists  should  be  sufficient  to  give  the  general  Idea. 

In  these  examples,  there  are  two  in^ortant  aspects:  the  overall 
sequence  of  the  skills  and  each  individual  physical  skill.  Since  success 
In  learning  a physical  skill  depends  on  both  the  learning  of  the  Indi- 
vidual physical  skills  and  the  overall  sequence  of  the  physical  skills, 
both  must  be  described. 


K 2,3.4  Perform  Learning  Analysis  for  the  Attitude  Category 


i 

The  analysis  in  the  category  of 
attitudes  is  more  difficult  since  It 
is  not  possible  to  observe  attitudes 
directly.  This  analysis  can  proceed 
by  asking  the  question,  "What  would 
a person  do  If  he  had  the  desired 
attitude?"  This  Is  done  by  writing 
down  the  things  you  would  expect  someone 
to  say  or  do  that  would  cause 

you  to  agree  that  he  had  the  dCiired  attitude.  The  purpose  of  the 
analysis  of  learning  attitudes  is  to  identify  the  specific  performances 
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which  mjst  be  treated.  To  perform  an  analysis  on  an  attitude,  ask: 
"Miat  choices  must  a person  make  that  would 
Indicate  the  desired  attitude?" 


EXAMPLE 


If  the  desired  attitude  was  that  one  has  an  "appreciation 
for  safety,"  further  analysis  may  Indicate  that  the 
behaviors  Involved  are  wearing  safety  equipment,  following 
safety  rules,  keeping  the  work-area  free  of  any  hazards, 
suggesting  possible  ways  to  Improve  the  safety  of  his  J(^, 
or  reporting  safety  hazards. 


The  analysis  of  attitudes  starts  with  the  abstract  Idea,  and  proceeds 
to  the  behaviors  tha»-  are  acceptable  as  evidence  of  that  attitude.  This 
Is  because  attitudes  are  difficult  If  not  impossible  to  observe. 


EXAMPLE 


If  the  desired  attitude  for  radio  operators  Is  a 
protective  attitude  towards  equipment,  the  question 
should  be  asked,  "What  does  an  operator  who  has  a 
protective  attitude  do?"  Some  possible  answers  are: 

1.  Does  not  let  the  battery  corrode. 

2.  Keeps  the  set  and  Its  components  free  of  dirt  and  dust. 

3.  Does  not  lose  any  of  the  components. 

4.  Performs  regular  maintenance. 

5.  Keeps  equipment  In  Its  carrying  case. 

First,  try  to  change  my  negative  statements  to  positive  ones. 
They  are  easier  to  observe; 

1.  Keeps  the  battery  and  battery  case  free  from  corrosion. 

2.  Keeps  the  components  properly  attached  and  stored. 

Next,  remove  duplications  and  Items  that  are  not  good  evidence. 
You  might  say  that  If  he  performs  regular  maintenance,  the  set 
and  Its  components  will  be  free  of  dirt  and  dust,  so  eliminate 
Iton  2,  Also,  keeping  equipment  in  Its  carrying  case  Is  recom- 
mended only  when  equipment  is  being  transported;  It  is  not  a 
^od  storage  place.  So  eliminate  5,  The  results  are: 
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1.  Keeps  the  battery  and  battery  case  free  from  corrosion. 

2.  Keeps  the  cony>onents  properly  attached  >ii  correctly 
stored. 

3.  Performs  regular  maintenance. 

These,  then,  are  acceptable  evidence  that  the  operator  protects 
his  equipment  and  they  are  tangible  things  upon  which  Instruc- 
tion can  be  developed. 

3.0  OUTPUTS 

The  outputs  of  this  block  should  consist  of: 

3. 1 Products 

1.  TLOs  for  tasks  selected  for  training.  These  terminal 
learning  objective  statements  will  Include  conditions 
and  standards.  (See  exan|>le,  page  31.) 

2.  LOs  and  LSs  resuiting  from  the  learning  analysis  of  the 
TLOs.  These  will  be  documented  In  a format  that  shows  the 
interrelationships  between  the  Items.  (See  Figure  II. 3, 
page  32,  and  example,  page  33, } 

3.2  Other  Documentation 

1.  A summary  statement  of  the  assumptions  made  about  student 
entry  behavior. 

2.  Any  revisions  to  JPMs  resulting  from  the  learning  analyses. 


Terminal  Learning  Objectives  for  Tasks  Performed 
by  OH-58  Helicopter  Repairman: 

1.  In  a maintenance  work  area  using  an  OH-58  helicopter, 
Technical  Manual  55-1520-228-20,  a general  mechanics 
tool  kit,  and  a maintenance  platform,  the  student  will 
Inspect  the  main  transmission  oil  systen  in  accordance 
with  Technical  Manual  55-1520-228-20. 

2.  The  stuctent  will  Install  a call  rotor  gear  box  In 
accordance  with  Technical  Manual  55-1520-228-20,  given 
a general  mechanics  tool  kit,  DA  Form  2408-13, 

Te^nical  Manual  55-1520-228-20,  an  OH-58  helicopter, 
and  a maintenance  platform  In  a maintenance  work  area, 

3.  Given  an  OH-58  helicopter.  Technical  Manual  55-1520- 
228-20,  DA  Form  2408-13.  nnd  a general  mechanics  tool 
kit,  the  student  will  maintain  the  power  train  system 
In  accordance  with  Technical  Manual  55-1520-228-20, 

In  a maintenance  work  area  with  a maintenance  platform. 


REMOVE  FAN  AND  SHAFT  ASSEMBLY 


/ SHIMS  \ 
'^BONDED  TO  ' 
HELICOPTER 
.STRUCTURE?/ 


MARK  OUTLINE  OF 
SHIMS  ON  HELICOPTER 


REMOVE  SHIMS 


(^STOP^ 


FIGURE  n 3:  Learning  Analysis  ol  the  Task;  REMOVE  MAIN  DRIVE  SHAFT  FROM  OH  50  HELICOPTER 
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LO  and  LSs  for  TLO  3: 

Laaming  Ctojective:  Given  an  W1-S8  helicopter. 
Technical  Manual  55-1520-228-20,  DA  Form  2408-13,  and 
a general  mechanics  tMl  kit,  the  stu(tent  will  remove 
the  main  drive  shaft  In  accordance  with  Tedmical 
Manual  55-1520-228-^. 

Learning  Steps: 

1.  .lemove  cooling  fan  and  shaft  ass^ly. 

2.  Remove  after  fairing. 

3.  Remove  forward  short  tall  rotor  dilve  shaft. 

4.  Ramove  aft  short  tall  rotor  drive  shaft. 

5.  Drain  oil  tank,  If  necessary. 

6.  Remove  engine  oil  cooler. 

7.  Remove  oil  fittings  and  lines. 

8.  Remove  flexible  air  duct. 

9.  Remove  fan  and  shaft  assembly. 

10.  Mark  outline  of  shims  on  helicopter,  If  necessary. 

Jl.  Remove  shims.  If  necessary. 
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INPUTS 


FIGURElr.1:  Flowchart  ol  BlockH.I:  DEVELOP  OBJECTIVES 


BLOCK  II. 2:  DEVELOP  TESTS* 
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OVERVIEW 


Testing  is  used  after  training  to  determine  whether  or  to  what 
degree  trainees  learned  what  they  were  intended  to  learn,  In  addition, 
entry  tests  are  used  to  reject  people  for  training  nr  to  prescribe 
remedial  training.  Pretests  that  match  the  posttests  are  used  for 
placement  within  a training  program  or  for  exempting  trainees  from  a 
program,  and  wi thin-course  tests  are  used  to  determine  trainee  progress 
and  need  for  assistance  during  the  training. 


•Portions  of  this  material  have  been  adapted  from;  Swezey,  R.  W. , and 
Pearlstein,  R.  B.  Developing  criterion  referenced  tests  (287-AR18(2)- 
IR-0974- RW5) , Res  ton,  Va. : Applied  Science  Associates,  September  L9!'4 
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DEVELOP  TESTS 
1.0  INTRODUCTION 

Each  terminal  learning  objective  (TLO)  is  derived  from  a JPM,  The 
training  test  for  the  TLO  may  be  identical  to  the  JPM,  or  there  may  be 
additional  constraints  in  the  training  setting  that  decrease  the  level 
of  fidelity  of  the  training  test.  The  change  may  be  iii  the  action,  the 
initiating  cues,  conditions,  or  standards.  Each  learning  objective  (LO) 
must  be  tested  in  the  training  setting.  For  the  same  reason  mentioned 
above,  these  tests  may  or  may  not  be  identical  to  related  parts  of  the 
JPM.  The  learning  steps  (LSs)  may  or  may  not  be  tested  individually; 
however,  they  are  often  tested  as  part  of  the  TLO  or  LO  test. 

In  the  learning  analysis  of  the  TLO,  many  LOs  and  LSs  may  be  listed. 
Some  of  these  are  steps  in  the  original  task  or  JPM;  however,  some  of 
them  represent  other  skills  or  knowledges  that  must  be  obtained  to  master 
the  TLO.  Generally,  these  other  skills  or  knowledges,  once  mastered,  are 
of  no  particular  further  concern  to  the  evaluators  (testers). 


EXAMPLE 


Analysis  of  the  skills  needed  for  multiplication  of 
three-digit  numbers  would  yield  an  LO  or  LS  stating 
an  "addition"  skill.  This  "addition"  skill  probably 
would  be  tested  in  an  entry  test,  diagnostic  pretest, 
and  within-course  test;  however,  it  would  not  be 
tested  separately  in  the  TLO  test  at  the  end  of 
training. 


There  are  often  many  differences  in  learning  a task  and  performing 
a task.  The  most  obvious  is  the  concentration  on  details  required  for 
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performing  even  a simple  action  for  the  first  time.  Each  step  is  con- 
sciously performed.  After  learning,  the  steps  become  more  antomatic. 

EXAMPLE 

1.  Disassemble  an  M16. 

2.  Follow  a pre-flight  ciiecklist. 

3.  Repack  a parachute. 

Another  difference  is  the  standard  of  performance.  During  training,  the 
time  required  to  perform  is  usually  longer,  the  degree  of  proficiency  is 
lower  and  the  product  is  less  than  perfect. 


EXAMPLE 

1.  Time  to  troubleshoot  an  engine  malfunction 

2.  Quality  of  a sheet  metal  joint 

3.  Amount  of  assistance  required  to  draw  the  accident 
scene  on  an  SF91  report 


The  inputs  to  this  block  are  the  tasks  selected  for  training,  along 
with  their  supporting  documentation,  the  JPMs  developed  in  Block  1.3, 
and  the  TLOs , LOs , and  LSs  derived  in  Block  II.l. 

The  procedures  for  developing  tests  are  shown  on  Figure  II. 4,  the 
fold-out  page  at  the  end  of  this  block. 

The  outputs  of  this  block  include  a posttest  (end  of  training  test), 
and  as  required,  an  entry  test,  pretests,  and  wi thin-course  tests. 
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2.0  PROCEDURES 


How  d^Ulled  and  Inclusive  a test  should  j ^ | 

be  depends  upon  the  use  for  which  It  Is  | . 

Intended.  1 

If  the  need  Is  to  place  trainees  In  I-  I— — I 

remedial  courses,  place  them  at  various  [ | | [ 

points  within  a course,  or  exempt  them  I LJ 

from  a course,  the  test  must  be  sufficiently 
broad  to  cover  all  the  course  prerequisites, 
entry  behavior,  and  all  the  TLOs  and  LOs . 

It  must  be  detailed  enough  to  discriminate  between  those  who  can  perform 
and  those  who  cannot  perform. 

If  the  test  Is  to  be  used  for  making  an  accept/reject  for  training 
decision,  It  should  be  a relatively  narrow  test  that  discriminates 
between  those  who  have  the  necessary  entry  behaviors  and  those  who  do 


2.1  Determine  How  Detailed  the  Test  Should  Be 


If  the  purpose  of  the  test  Is  to  give  trainees  information  on  how 
they  are  progressing,  the  test  should  Include  test  items  on  TLOs,  LOs, 
and  possibly  LSs  that  can  be  scored  either  mechanically  or  by  the  student 
to  give  him  feedback.  This  feedback  can  also  be  used  to  recycle  the 
student,  prescribe  remedial  instruction,  or  place  the  trainee  ahead. 

When  a course  is  being  developed  there  Is  an  additional  use  for  test 
results.  If  students  are  tested  frequently  the  scores  provide  a record 


of  how  well  the  instruction  Is  performing.  Errors  by  students  pinpoint 
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fahures  In  the  Instruction.  Early  achievement  of  goals  or  higher  than 
expected  standards  are  Indications  that  the  Instruction  should  be 
shortened.  Testing  for  validation  (Block  III. 5)  should  be  detailed  and 
should  have  a detailed  scoring  procedure. 

2.2  Translate  TLOs  and  LOs  Into  Test  Items 


i 


(• 


I 


2.2.1  Testing  Mortal  Skills 

Mental  skills  can  often  be  tested  with 
paper  and  pencil  tests  even  when  the  job 
task  Is  not  a paper  and  pencil  task. 


EXAMPLES 

LEARNING  OBJECTIVE 
ACTION  STATEMENT 

1.  Select  the  proper  fire  ex- 
tinguisher for  the  following 
classes : 

a.  ordinary  corrbustible 
materials 

b.  flammable  liquids 

c.  electrical  fires 


2.  Identify  US  and  USSR 
destroyers. 


TEST  ITEM 


Select  the  proper  fire  ex- 
tinguisher for  the  following 
fires : 

Fires-- 
paper 
wood 
rags 
trash 
grease 
flanomable 
liquid 
electrical 

(Using  p!  cures)  classify  the 
follc'-'’ :ig  ships  as  US  or  USSR: 

Claud  Jones 

Forrest  Sherman 

Kola 

Riga 

Kotlin 


Extinquishers- 

water 

soda  and  acid 

foam 

CO2 

dry  powder 
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LEADING  OBJECTIVE 

ACTION  STATEMEWT  TEST  ITEM 


3.  Identify  medical  service 
symbols. 


4.  Visually  detect  camouflaged 
armored  vehicles  In  a terrain 
setting. 


5.  Choose  a diagnostic  strategy 
for  a malfunction  In  ?:he 
nain  drive  shaft  of  an  OJI-58, 


Which  of  these  are  medical 
service  syntols? 

□ □ n □ □ 

A B C 0 E 

Hark  the  location  of  armored 
vehicles  In  the  following 
slides  If  any  are  In  the  picture. 

□ □ □ 

ABC 

Using  the  simulator  for  each 
preprogrammed  malfunction, 
choose  a diagnostic  strategy 
for  locating  the  problem. 


In  each  case  the  answers  could  be  written.  For  the  same  skills, 
there  could  also  be  performance  test  Items. 

The  disadvantage  of  the  written  test  Items  Is  the  artificial  Initiation 
and  temlnatlon  cues.  For  Instance,  the  real  cue  to  select  a fire  extin- 
guisher Is  the  existence  of  a fire.  This  is  a cue  to  determine  its  class 
then  select  tlie  fire  extinguisher.  Similar  real  cues  exist  for  classifying 
ships,  identifying  symbols,  and  detecting  camouflaged  vehicles.  Rather 
sophisticated  simulation  of  field  exercises  would  be  required  to  provide 
surprise  initiating  cues. 

In  addition,  each  of  these  mental  activities,  such  as  classify,  identify, 
and  detect.  Is  an  initiating  cue  to  some  action:  Put  out  the  fire;  decide 
whether  a ship  is  friendly  and  take  appropriate  action;  decide  whether  to 
bomb  or  not  bomb  a building  and  do  It;  detect,  identify  and  destroy  enemy 
armored  vehicles. 


U 


> 

i 

i 


j 


j 

i 


5 

» 
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There  are  mental  skills  objectives  for  which  there  are  a large  number 
of  Items  and  those  with  relatively  few. 

Rather  than  having  degrees  of  proficiency  like  many  physical  skills, 
mental  skills  are  supposed  to  be  either  learned  or  not  learned,  SoiMtImes 
test  scores  for  many  people  are  much  alike,  sometimes  the  scores  clearly 
divide  the  people  Into  two  groups,  and  sometimes  the  scores  are  spread 
rather  evenly  over  a wide  range. 


EXAMPLES 

1.  In  the  first  case  If  the  results  were  drawn  for 
a five  Item  test,  they  would  look  like  this: 

X 
X 
X 
X 

XXX 
XXX  X X 


SCORE 

In  this  case,  67%  of  the  people  scored  4 or  5. 

The  conclusion  drawn  Is  probably  that  those  67% 
have  mastered  the  ..kill  and  the  other  33%  have  not. 

2,  In  the  second  case,  if  the  results  were  drawn  for 
a five  item  test,  the  results  would  look  like  this: 


Q. 

0 

01 
a. 


o 

=«t= 


X 

X 

X 

X 

X 


1? 


X 


T T~ 

SCORE 


X 

X 

X 

X 

X 


This  case  clearly  divides  people  into  "go"  and 
"no-go"  groups.  Both  of  the  above  are  reasonable 
patterns  to  get  when  testing  mental  skills. 


SCORt 


X 

X 


1 ^ 


If  results  look  like  this,  check  to  see  if  the 
Items  are  all  testing  the  same  mental  skill  and  if 
there  are  other  factors,  like  guessing,  that  could 
be  influencing  the  scores. 


2.2.2  Testing  Information 

Information  test  items  are  relatively  easy  to  develop  from  the 
learning  objectives. 


EXAMPLE 


LEARNING  OBJECTIVE 

ACTION  STATEMENT  TEST  ITEM 


Describe  the  rules  of  war 
including  forbidden  targets, 
tactics,  and  techniques; 
treatment  of  captives, 
detainees,  civilians  and 
private  property;  and  what 
to  do  when  crimes  are 
committed. 


Describe  the  rules  of  war: 
include  forbidden  targets, 
tactics,  and  techniques; 
treatment  of  captives, 
detainees,  civilians  and 
private  property;  and  what 
to  do  when  crimes  are 
committed. 


The  problems  with  testing  information  are  in  setting  standards  when 
there  is  no  Job  criterion  and  sampling  when  there  is  too  much  information 
to  be  tested.  In  both  cases  the  test  developer  makes  an  arbitrary  deci- 
sion, Either  he  makes  the  decision  himself  or  he  consults  "experts"  and 


i 


j 


I 


I 


L 
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gets  a consensus,  htost  Information  Is  subordinate  to  other  learning  or 
to  otlier  performances.  Although  an  LO  that  tests  the  Information  must 
be  made.  It  m<(y  not  be  Included  In  the  TLO. 


EXAMPLE 


The  action  part  of  a TLO  for  Radio  Teletypewriter 
Comnuril cation  Procedure  Is:  Ar.-ange  the  contents  of 
a DO  form  173  into  the  correct  radio  teletypewriter 
...dssagc  Tormat.. 

The  LOs  Include:  Listing  the  parts  of  a mei:»oe,  the 

components  of  the  message  heading  and  the  (detalis 
omitted  for  simplicity).  Te.it  Items  on  the  LOs  within 
the  course  require  the  trainee  to  list  these  parts. 

The  TLO  test  item  assumes  that  if  the  trainee  can 
identify  the  errors  and  send  the  right  error  message 
he  also  knows  the  parts  of  the  message,  heading  and 
address,  etc, 

LO  test  items: 

1.  List  the  three  parts  of  a message. 

a.  

b.  2__ 

c.  


2.  A message  heading  may  contain  four  components. 
List  them  in  proper  order. 

a.  

b.  j; 

c. 

d.  

NOTE:  Complete  statements  3 through  6 by  filling  in 

the  blanks. 

3.  The  call  and  transmission  Instructions  appear  in 

the  component  of  the 

message  heading. 

4.  The  precedence,  date- time-group  and  message 

instructions  appear  in  the  

component. 

5.  The  accounting  information  and  group  designation 

appear  in  the component. 
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6.  The  originator,  action.  Information  and  exempted 

addresses  appear  In  the canponent. 

7.  List  the  four  prosigns  which  could  appear  In  the 
address  component. 

a.  

b.  

c.  

d.  


2.2.3  Testing  Physical  Skills 

When  the  JPMs  have  high  job  fidelity,  the  training  tests  should  follow 
as  closely  as  possible.  The  problems  are  likely  to  be  In  Initiating  and 
terminating  cues  and  standards.  TlO  test  standards  must  be  set  to  match 
the  JPM.  Within  course  tests  have  to  reflect  realistic  degrees  of  pro- 
ficiency after  each  amount  of  training.  This  can  be  based  on  past 
experience  or  empirically  developed  with  a new  training  program. 


EXAMPLE 


LEARNING  LEARNING  OBJECTIVE 

SUB-CATEGORY  ACTION  STATEMENT 


TEST  ITEM 


Performing  gross  Load  the  chaparral 
.rxjtor  skills  (7) 


Load  the  chaparral 
(Checklist) 


Steering  and  Guide  the  TOW  missile  to 

guiding— contin-  a selected  target, 
uous  movement  (8) 


Guide  the  TOW  missile 
to  a selected  target. 
(Checklist:  number 
of  hits) 


Positioning  move-  Asseir^le  and  disassemble 
ment  and  recalling  the  M203  grenade  launcher, 
procedures  (9) 


Assemble  and  disassemble 
the  M203  grenade 
launcher.  (Checkl  st) 


Voice  communi-  Cotimunicate  with  divers 
eating  (10)  at  a depth  of  30  meters 

using  telephone  equip- 
ment. 


Send  and  re<:3ive 
messages  of  at  least 
two  minutes  duration 
with  divers  30  meters 
below  the  surface. 
(Checkl  i?r.) 
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The  Initiating  cues  may  be  non-existont  or  considerably  different 
at  the  beginning  of  training  and  added  or  changed  In  the  final  stages. 


EXAff’LE 


Learning  objective  In  maiksmanshlp-- 

Actlon  statement:  Hit  a target  at  "x"  meters. 
Beginning  Initiating  cues:  When  Instructor  says, 

"fire,"  fire. 

Conditions:  Official  paper  target  with  concentric 
circles. 

Standards:  Score  above  XX. 

The  JPM  from  wtdch  the  marksmanship  LO  was  derived 
might  have  Included  the  Initiating  cues  and  terget 
of  "punchy  Pete,"  and  standards  of  "hit  so  that 
punchy  Pete  falls  down." 


2.2.4  Testing  Attitudes 

How  the  trainee  feels  about  something  cannot  be  measured  directly. 
While  he  can  be  asked  directly,  he  may  or  may  not  tell  his  real  feelings. 
But  what  is  more  important  Is  how  he  behaves . When  a learning  cnalysif 
was  done  of  <n  attitude,  items  were  listed  that  ’..'ould  be  acceptable  as 
evidence  that  the  desired  attitude  had  been  learned.  This  evidence 
must  be  translated  into  a tost  or  a checklist. 


EXAMPLE 


Learning  Objective:  Radio  operator  will  have  a 

protective  attitude  towards  his  equipment: 

1.  he  keeps  battery  and  battery  case  free  from 
corrosion 

2.  He  keeps  the  components  properly  attached  or 
correctly  stored. 

3.  He  performs  regular-  maintenance. 
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Test  Item  possiblli tl^i; 

A.  Ask  the  trainee 

1.  Do  you  protect  your  equipment? 

Yes  No 

2.  Do  you  keep  the  battery  and  battery  case 
free  from  corrosion? 

Always  Usually  Rarely 

3.  Do  you  keep  the  components  properly  attached 
or  cor»^ctly  stored? 

Always  Usually  Rarely 

4.  Do  you  perform  regular  maintenance? 

Always  Usually  Rarely 

8.  Supervisor  observes  and  scores  trainee 

1.  Are  his  battery  and  battery  case  free  from 
corrosion? 

Yes  No 

2.  Are  the  con^onents  properly  attached  or 
correctly  stored? 

Yes  No 

3.  Mas  regular  maintenance  been  performed? 

Yes  No 

The  answers  in  "A"  will  probably  be  influenced  by  the 
trainee's  perception  of  what  you  want  him  to  answer. 
The  answers  in  "B"  reflect  the  condition  of  his  equip 
ment  at  a point  in  time  and  the  presence  of  evidence 
selected  on  which  to  judge  his  attitude. 
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2.3  Set  Training  Standards 

Wien  the  trainee  Is  on  the  Job,  his 
standard  should  equal  the  Job  standard. 

From  tiie  bealnning  of  training  until  he 
Is  on  the  Job,  the  standards  may  be 
different  fr«n  the  job  standard.  There 
may  be  a gradual  increase  in  t.he  standard 
through  training,  it  may  even  reach  a 
level  higher  than  that  required  on  the  Job 
to  allow  for  some  skill  decay  during  the 
lag  between  training  time  and  Job  pei formance. 

Some  Job  standards,  such  as  those  for  using  hand  grenades,  are  very 
specific  and  cannot  be  changed.  But,  for  training,  duimy  grenades  could 
be  used  until  those  standards  are  niet. 

The  end-of-tra1n1ng  standard  should  approximate  the  JPM  standard. 
Within-training  standards  should  be  based  on  reasonable  expectations 
after  given  amounts  of  time  In  training.  They  should  be  low  enough  to  be 
attainable  and  high  enough  to  be  an  Interesting  challenge  to  the  trainees. 

EXAMPLE 

At  the  e;id  of  two  weeks  of  senior  life  saving 
instruction,  a physically  Fit  graduate  must  swim 
one  mile.  The  standard  for  the  end  of  the  third 
djo'  might  be  100  yards. 

At  times  there  are  no  real  Job  standards  and  the  TLO  test  standards 
should  match  the  JPM  standards  as  closely  as  practical. 
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EXAMPLE 


Task:  In  comhat,  observe  the  cjstcms  and  tfie  laws  of 

war  as  presented  in  the  Geneva  Convention  and  Hague 
Regulations  In  the  following  areas: 

A.  Forbidden  targets,  tactics  and  techniques 

1.  Do  not  attack  noncoirbatants 

2.  Preserve  cultural  and  hurnani tarlan  property 

3.  Protect  medical  personnel  and  facilities 

4.  Do  not  hide  behind  the  medical  service 
symbols. 

B.  Enemy  captives  and  detainees 

1.  Let  enemy  soldiers  surrender. 


JPM:  In  a set  of  simulated  field  exercise  problems, 

observe  the  customs  and  laws  of  war  as  presented  in 
the  Geneva  Convention  and  Hague  Regulations  with  no 
more  than  one  error  in  six  problems  In  the  following 
areas : 

A.  Forbidden  targets,  tactics  and  techniques 

1.  Do  not  attack  noncombatants 

2.  Preserve  cultural  and  humanitarian  property. 
. (same  as  job  task  above) 


In  some  training  settings,  simulated  field  exercises  may  not  be 
practical.  The  TLO  may  have  a low  physical  fidelity  to  the  task/JPM. 


EXAMPLE 

TLO: 

Action:  Apply  the  customs  and  laws  of  v/ar  as 
presented  in  the  Geneva  Convention  and  Hague 
Regulations  in  the  following  areas: 

A.  Forbidden  targets,  tactics  and  techniques 

1.  Do  not  attack  noncombatants 

2,  Preserve  cultural  and  humanitarian  property. 

4,  Do  not  hide  behind  the  medical  service  syr.ibols. 
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II 


Condition:  Given  a written  set  of  simulated  problene. 

Standard:  No  more  than  one  error  In  six  problems. 

TLO  Test:  (A4  only)  Do  not  hide  behind  the  medical 

service  symbols. 

Tost  item: 

You  are  in  a village  and  have  been  cut  off  from  your 
squad.  Wiilch  buildlng(s)  may  you  use  for  cover? 


In  a war,  the  Job  task  standard  would  be  that 
combatants  are  not  supposed  to  violate  any  laws  of 
war.  In  both  the  JPM  and  the  TLO  test,  the  task 
standard  is  not  applicable.  Since  the  TLO  test  is 
a written  exercise,  there  is  no  external  criterion; 
therefore  set  an  artitrary  standard. 


For  many  LOs  there  will  be  no  external  criterion. 


EXAMPLE 


Learning  Analysis  of  A4 


LOl  test:  Which  of  these  symbols  are  medical 

service  symbols? 


^ ^ + C ® « 
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To  set  the  standards  for  this  LO,  you  may  decide 
whether  all  medical  service  symbols  must  be  identi- 
fied as  such  or  if  it  is  acceptable  for  the  learner 
to  select  A and  F which  are  not  medical  service 
symbols.  While  the  latter  is  an  error,  it  Is  one 
that  does  no  harm. 

Standards  may  also  be  set  by  consensus.  There  is  no  known  job 

standard  for  LOs  such  as,  "An  E~2  should  be  able  to  recall  the  Code 

« 

of  Military  Justice,"  When  the  question  is  "how  much"  should  he  be 
able  to  recall,  the  usual  feeling  is  that  "more  is  better."  A standard 
can  be  set  for  this  by  consensus.  Have  experts  or  supervisors  specify 
just  what  an  E-2  needs  to  be  able  to  recall.  Or,  test  E-2s  who  are 
identified  as  knowing  what  they  need  to  know  and  use  their  scores  as 
a standard.  Neither  of  these  methods  is  very  satisfactory  and  should 
only  be  used  cs  a last  resort. 

2.3.1  False  Positives  and  False  Negatives 

Cne  purpose  of  testing  is  to  separate  those  who  can  pertorm  the 
learning  objectives  from  those  who  cannot.  Since  tests  are  not  perfect-- 
there  are  external  factors  that  cannot  be  control  led j the  trainers  may 
have  a bod  day  or  the  test  monitor  or  scorer  may  make  a mistake — people 
are  sometiires  misclassified.  If  a person  cannot  perform  the  task  but 
he  passts  the  test,  he  is  called  a false  positive.  If  he  can  perform 
the  task  but  fails  the  test,  he  is  called  a false  negative.  Figure  II. 5 
shows  these  classifications  along  with  the  "correct"  classifications. 
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pass  test 

fail  test 

can  do  task 

correct 

false 

negative 

cannot  do  task 

false 

correct 

posi  tive 

FIGUfiE  II. 5:  False  PosiMves  and  False  Negatives 


The  goal  is  to  develop  tests  and  scoring  procedures  to  make  the  fewest 
possible  misclassifications.  However,  there  are  times  when  one  of  the 
errors  is  more  serious  than  the  other. 


EXAMPLE 


In  the  case  of  a dangerous  task  such  as  mine 
sweeping,  it  would  be  less  costly  in  terms  of 
risk  to  people  and  property  to  fail  a trainee 
who  could  adequately  perform  the  job  (therefore, 
making  him  a false  negative),  than  it  would  be 
to  pass  someone  who  real ly  could  not  perform 
the  task  (a  false  positive). 


I 

1 
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Sometimes  a change  in  standard  will  reclassify  trainees  from  a fail 
to  a pass. 


EXAMPLE 


A trainee  cook  may  fail  the  test  because  he  is 
slow.  With  a little  time  on  the  Job,  he  would 

If  cooks  are  desperately 
to  change  the  test  standard. 


be  an  adequate  cook, 
needed,  you  may  wish 


J 


itLiiimii 
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Set  the  test  standards  to  minimize  both  false  positives  and  false 
negatives.  If  there  is  a risk  or  price  attached  to  one  or  the  other, 
minimize  the  one  with  the  high  risk  or  high  cost  associated  with  it, 
at  the  expense  of  the  other. 

2.4  Rank  Order  Students  (If  Required) 

Tests  associated  with  tne  ISD  process 
are  primarily  intended  to  indicate  whether 
or  not  a student  has  achieved  a specified 
level  of  mastery.  Therefore,  emphasis  has 
been  on  procedures  for  constructing  tests 
v/hich  accurately  indicate  studenc  performance 
of  carefully  defined  behaviors.  However, 
sometimes  there  is  a eed  to  assign  ranks  to 
students.  That  is,  sometimes  the  top  S%  of  your 

graduates  must  be  identified,  or  the  students  listed  in  order  of  their 
ach ■' evemeri t , or  some  'ther  type  of  report  must  be  presented  which  re- 
flects an  ordering  of  the  students.  If  JPMs  have  been  used  to  develop 
the  tests,  the  tests  and  scoring  procedures  developed  may  not  allow  for 
efficient  rank  ordering. 

For  example,  if  the  test  has  four  out  of  five  correct  as  its  passing 
score  and  the  entire  class  passes  the  test,  there  is  no  basis  for  rank 
ordering  tne  students.  There  are  alternate  procedures  that  can  be  used 
to  obtain  ordered  student  data-  however,  a most  important  point  is  that 
tests  are  designed  to  do  something,  and  if  they  are  used  inappropriately , 
they  may  not  work  well.  So,  if  measuring  absolute  performance  is  impor- 
tant, design  a test  to  do  that,  but  if  rark  ordering  students  is  important, 
design  a test  to  do  that. 


[ J ! r ' 
iip-!  7^^ 
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Good  rank  order  tests  have  many  of  the  same  characterlstfcs  as  tests 
designed  to  Indicate  a specific  set  of  abilities.  For  example,  con- 
struction of  rank  order  tests  requires  the  same  definition  of  what  is  to 
be  tested  and  the  same  concern  for  the  mechanics  of  test  directions,  test 
administration,  scoring,  and  the  like.  The  concepts  of  test  reliability 
and  validity  must  be  maintained.  Rank  order  tests  are  different  in  that 
they  .ire  designed  to  produce  a distributior  of  scores;  that  is,  if  ten 
individuals  take  the  test,  there  likely  will  be  ten  different  scores. 

Rank  ordering  is  appropriate  when  there  is  a need  to  select  the 
fastest,  the  most,  accurate , or  the  best  producer. 

EXAMPLES 

1.  If  you  need  the  best  marksmen,  ranx  order  the  trainees 
on  the  basis  of  their  marksmanship  test  or  on  punchy 
Pete. 

?.  If  you  want  the  best  golfers,  rank  them  from  low  to 
high  scores. 

3.  If  you  want  the  best  runners,  rank  them  from  shortest 
to  longest  tines. 

4.  If  you  want  tie  best  radio  teletype  operators,  rank 
them  on  the  basis  of  speed  and  accuracy. 

In  each  of  the  above  ca^es  scores  can  be  assigned  to  the  performance. 
In  some  cases  there  is  an  existing  criterion.  It  may  be  that  radio 
teletype  operator  trainees  who  cannot  type  at  20  words  per  minute  at  the 
end  of  two  weeks  will  not  reach  the  riiinimum  acceptable  level  by  the  end 
of  training  and  those  who  type  faster  will  be  better  on  the  job.  If 
this  were  so  you  would  have  a criterion. 


In  other  cases  there  ts  no  discemable  criterion. 


EXAMPLE 


Universal  Code  of 
Military  Justice's  major  25  points  was  given 
to  everyone  in  the  Armed  Forces.  While  it  is 
probably  safe  to  s^y  that 
the  distribution  of  scores  might  look  like  this; 

Score  Frequency 

25  X 

24  XX 

23  XXX 

22  XXX 

21  XXXX 

20  XXXXX 

19  XXXXXX 

18  XXXXXXX 

17  XXXXXXXXX 

16  XXXXXXXXXXX 

15  XXXXXXXXXXXX 

14  XXXXXXXXXXXKX 

13  XXXXXXXXXXXitXX 

12  XXXXXXXXXXXXX 

11  XXXXXXXXXXXX 

10  XXXXXXXXXXX 

9 XXXXXXXXX 

8 XXXXXXX 

7 XXXXXX 

6 XXXXX 

5 XXXX 

4 XXX 

3 XXX 

2 XX 

1 X 

How  many  questions  does  a military  person  need  to 

a<ivance  In  rank,  have 

a su.ceSjful  career,  perform  his  primary  mission  or 
^ a credit  to  the  service?  This  is  one  of  thwe’ 
ess  situations  in  which  it  is 
toting  to  say.  More!  They  should  know  more."  If 

to'tKrSe'K^iLr 
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While  It  Is  possible  to  express  scores  on  this 
test  as  the  percentages  of  correct  responses  out 
of  the  total  number  of  possible  responses,  and 
to  decide,  for  example,  that  everyone  who  gets 
75*  passes,  this  cut-off  score  Is  arbitrary  since 
U Is  not  based  on  any  established  criterion.  It 
is  also  possible  to  rank  order  the  students  and 
pass  the  top  75*,  This  alternative  Is  as  good-- 
or  bad--as  the  first  since  It  Is  not  known  how 
high  the  scores  should  be.  Being  arbitrary  by 
any  other  name  Is  still  being  arbitrary,  and 
arbitrary  criteria  should  be  recognized  as  such. 


2.5  Set  Cut-Off  Scores 

When  a cut-off  score  Is  set,  such  as 
70  percent  or  8 out  of  10  questions 
correct,  how  many  items  are  correct 
must  be  related  to  which  Items  were 
correct.  Often,  getting  LOs  correct  does 
not  guarantee  getting  the  TLO  correct,  and 
the  TLO  is  what  counts.  Also,  the  TLO  may 
include  the  LOs . 

EXAMPLE 

Test  item  1-LOl:  List  the  three  parts  of  a message. 

Test  item  2-L02: 

Test  item  9-L09;  List  the  prosign  meaning  "break" 
that  is  used  on  the  separation  lines  on  the  "adio 
teletypewriter  message  form. 

Test  item  lO-TLO:  Transfe.'  information  fi'om  the 
joint  message  form  {DD  Form  173)  to  a blank  sheet  of 
paper  and  arrange  the  content  of  the  message  heading 
in  the  proper  format  for  transmission. 
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The  TLO  and  Its  test  item  match  the  JPM.  Only  If 
the  student  can  pass  the  TLO  test  Item  can  he  do 
the  job.  Reaching  an  8 out  of  10  criterion  on  the 
10  Item  test  Is  meaningless  If  the  TLO  Item  Is  not 
one  of  those  done  cort«rt1y.  Only  a pass  on  the 
TLO  test  Item  means  "go."  The  scores.  In  this 
case,  on  the  LO  test  Items  can  be  used  to  help  find 
where  the  trainees  had  prob'ems  In  the  lesson  so 
remedial  Instruction  can  be  provided  or  the  Instruction 
can  be  revised.  The  LO  scores  should  not  be  used  for 
a "go/no-go"  decision. 


2.6  Detcmilne  Scoring  Procedures 

Once  all  the  test  Items  are  completed, 
the  ncKt  step  Is  to  carefully  document  the 
prc^cedures  for  administering  the  tests  and 
scoring  the  tests.  Any  special  equipment, 
facilities,  test  administrator  skills,  or 
other  resources  must  be  listed. 


2.7  Collect  Basel  in J Data 

if  there  is  existing  instruction  for 
the  DOS,  the  TLO  tests  developed  in  this 
block  should  bc^  adnnnistered  to  all  or  a 
large  sample  of  individuals  who  have  just 
completed  the  existing  Instpjctlon.  This 
will  serve  two  purposes. 
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1.  It  will  provide  an  opportunity  to  try  out  the  test  itens 
and  administration  procedures.  If  there  are  obvious 
problems,  these  can  be  used  as  a basis  for  making  revi- 
sions in  the  test  itons  or  the  administration  procedures. 

2.  It  will  provide  baseline  data  that  can  be  used  in 
Block  V.l  for  conducting  internal  evaluation. 

Badcground  and  training  information  should  be  collected  on  graduates 
used  in  the  test  trial,  since  wherever  possible,  these  same  graduates 
will  be  included  in  the  later  external  evaluation, 

3.0  OUTPUTS 

The  outputs  of  this  block  should  be: 

3.1  Products 

1.  Test  items  for  each  TLO.  LO  and  LS.  (The  LS  tests 
will  often  be  included  in  the  LO  tests.) 

2.  Test  items  organized  into  entry  tests,  pretests,  within- 
course  tests,  and  posttests.  (Finalizing  these  tests 

must  be  accomplished  after  structuring  of  learning  objectives 
in  Block  II .4. ) 

3.2  Other  Documentation 

1.  A summary  statement  of  baseline  data  from  test  trials. 

2.  Any  critical  decisions  made  in  accomplishing  the  above 
and  the  rationale  for  the  decision;. 
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EXAMPLES 

la.  Performance  test  or  real  problem  requiring  signalman  to 
pick  a draft  from  a hold,  or  between  deck,  and  deposit 

it  on  the  pier.  Present  slimilated  loading  problem  (number 
and  size  of  the  cargo  should  be  listed  for  the  test 
administrator.)  Student  must  direct  winch  operator  using 
the  correct  hand  signals.  (Time  limits  and  other  standards 
will  be  determined  by  the  problem) 

lb.  Information  test: 

LO:  Give  and  respond  to  correct  hand  signals  for 
operation  of  winches  for:  Lower  the  hook 

Raise  the  hook 


Hold  everything 

An  entry  test  might  Include  recognition  of  the  hand 
signals  such  as:  Match  the  hand  signals  with  the 
message  by  printing  the  correct  letter  by  each  picture— 

A.  Lower  the  hook 

B.  Stop 

C.  Lower  the  boom 


Students  who  passed  the  information 
test  would  then  oe  given  the 
performance  test. 


starting  from  a change  in  a job  (Recruiter  whose 
duties  have  changed  due  to  the  expiration  of  the 
draft),  this  example  sauries  from  a stue(y  done  for 
the  Arnty  Recruiting  Command.*  Job  analyses  were 
done  for  the  Recruiter,  Recruiting  Manager,  and 
a11  the  Jobs  through  the  comand. 

Recruiting  Managei — 

Goal:  Perform  supervisory  duties, 

TLO:  Review  follow-up  procedures  on  all  applicants 
processed  by  the  AFEES.  Evaluate  Individual 
recruiter's  follow-up  technique.  Given  case 
studies  of  follow-up  techniques  employed  by  each 
assigned  recruiter  and  corresponding  completed 
prospect  cards  with  varied  disposition  entlrles, 
e.g.,  enlisted,  qualified  not  enlisted  (QNE)  and 
disqualified  (mental -temporary , mental -permanent, 
physical -temporary,  physical -permanent,  and 
moral),  the  student  will  evaluate  each  recruiter's 
follow-up  technique  to  Identify  deficiencies. 
Acceptable  performance  will  Include  correctly 
Identifying  all  deficiencies  In  each  recruiter's 
follow-up  technique. 

Test  Items:  (What  must  the  learner  be  able  to  do  to 
perform  the  task?  Or,  what  knowledge  and  skills 
will  enable  the  learner  to  perform?) 

a.  Describe  the  purposes  of  follow-up 
(Attachment  #1) 

b.  Describe  follow-up  techniques: 

1)  Letters 

2)  Phone  calls 

3)  Making  or  Inviting  visits 

c.  Describe  techniques  for  counseling  applicants 
disqualified  at  the  AFEES: 

Given  a portrayal  of  an  applicant  who  has  been 
medically  or  mentally  disqualified,  the 
recruiter  will : 

1)  Make  a statement  of  understanding  or  sympathy 
concerning  the  applicant's  disqualification. 


♦This  has  been  adapted  from;  Sugrue,  £.  D.  (Project  Director).  Turning 
point:  fif)  evaluation  of  the  professional  skills  needed  to  support  the 
continued  growth  of  the  Volunteer  Army.  Boston,  Mass.:  Harbridge  House, 
Inc.,  19747 


60 


2)  Discuss  the  applicant's  mental  test  score 
compared  to  the  minimum  acceptable  (If 
applicable). 

3)  Discuss  the  probabilities  for  a (mental) 
retest  (If  applicable). 

4)  Sufimarlze  the  medical  disqualification 
(If  applicable) . 

5)  Explain  possible  reiredlal  action,  if  any 
(If  applicable) . 

Attachment  #1-- 

1.  Enlisted: 

a.  Identification  of  new  leads  (Individual  becomes 
a center  of  influence  (Cl)). 

b.  Enhance  favorable  Anny  image. 

1)  Insure  Individual  gets  what  he  was  guaranteed 
(option,  assignment). 

2)  Offer  asssltance  during  his  basic  and  advanced 
training  phases. 

2.  Qualified,  not  enlisted  (QNE); 

a.  Identification  of  new  leads. 

b.  Determine  reason  for  not  enlisting. 

c.  Get  the  Individual  to  reconsider  (react  to  reason). 

3.  Disqualified; 

a.  Identification  of  new  leads. 

b.  Eliminate  negative  opinions. 

c.  If  disqualification  is  tei.porary,  assist  the 
individual  in  attempting  to  eliminate  the 
disqualification, 

d.  Maintain  contact  in  the  event  regulatory  changes 
eliminate  the  disquall fier. 


NOTii:  Individual  disqualified  for  moral  reasons  (e.g., 
police  record)  are  not  normally  a subject  of  follow-up 
actions , 


FIG0REjr.4.  Flowchart  of  Blockl.?;  DEVELOP  TESTS 


BLOCK  II. 3:  DESCRIBE  ENTRY  BEHAVIOR 


p: 
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OVERVIEW 

A learning  analysis  was  performed  on  each  task  in  Block  II. 1 
to  the  point  of  assumed  entry  behavior.  It  is  time  to  test  these  assump- 
tions, and  make  any  necessary  corrections.  The  entry  behavior  of  the 
students  defines  the  limits  of  the  instniction  and  any  required  remedial  or 
preparatory  lessons.  A decision  is  made  as  to  whether  a pretest  will  be 
used  to  gate  students  around  any  of  the  instruction  that  they  have  already 
mastered.  If  this  decision  is  affirmative,  the  pretest  is  developed  her^». 


3est  Available  Copy 
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DESCP.IBt  ENTRY  BEHAVIOR 
l.Q  INTRODUCTION 

Entry  behavior  Includes  the  skills,  infonpatlon,  and  attitudes  that 
an  Individual  possesses  at  the  time  he  comes  for  Instruction.  An  Instruc- 
tional program  as.siaies  that  students  who  begin  instruction  will  already  have 
certain  skills,  knowledge,  and  attitudes.  The  primary  purpose  of  this 
block  Is  to  establish  exactly  what  these  entry  level  behaviors  are.  One 
goal  of  efficient,  effective  instruction  is  to  base  new  learning  on  what 
the  students  already  have,  and  to  avoid  having  them  waste  time  on  objec- 
tives they  have  already  mastered. 

Since  the  entry  behaviors  of  individuals  may  vary  greatly,  it  is  not 
always  practical  to  start  instruction  at  th??  point  that  matches  the  behavi- 
ors of  al_^  prospect! ve  students.  This  means  some  individuals  should  either 
be  rejected  or  given  remedial  lessons. 

Usually  the  entry  level  is  set  at  a point  whe"e  most  prospective  students 
have  the  prerequisite  skills,  knowledge,  and  attributes.  In  Block  II. 1, 
you  made  assunpti ons  about  these  entry  behaviors.  Now  it  is  time  to 
test  those  assumptions  by  measuring  the  performance  of  a group  of  inai- 
viduals  who  are  representative  of  trainees  who  will  take  the  instnxtioii 
after  it  is  developed. 

The  assumptions  made  about  student  entry  behaviir  were  based  on 
familiarity  with  the  general  level  of  skills,  knowledge,  and  attitude  of 
trainees  adnitted  to  the  particular  DOS,  and  on  extern’!  systems  require- 
ments. These  external  requirements  are  statements  of  rrUiimum  entry  behaviors 
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usually  established  by  the  personnel  system.  These  requirements  exclude 
from  all  or  certain  courses  of  instruction  individuals  who  do  not  meet 
minimum  service-wide  or  specific-assignment  requirements.  Some  of  these 
external  requirements  are  as  follows: 

1.  Administrative  requirements  which  involve  personnel  compliance 
with  regulations  and  states.  A trainee's  rank,  security 
clearance,  time  remaining  in  the  service,  or  past  criminal 
record  may  exclude  him  from  certain  types  of  training, 

2.  Physical  requirements  which  include  specific  skills  and  general 
fitness  which  may  include  age,  height,  color  perception, 
night  vision,  etc. 

3.  Academic  requirements  which  include  educational  or  aptitude  con- 
siderations, specific  courses,  acadenrc  degrees,  foreign 
language  fluency,  etc. 

The  inputs  to  this  block  are  the  learning  objectives  and  tests  developed 
in  Blocks  il.l  and  Ii.2,  and  a statement  of  the  external  entry  behavior 
requirements.  The  procedures  are  shown  on  the  flowchart  in  Figure  II. 6 
(the  fold-out  page  at  the  end  of  this  block).  The  outputs  will  include  a 
revised  or  verified  list  of  learning  objectives  and  their  tests,  an 
entry  test  if  required,  and  a pretest  if  required. 
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2.0  PROCEDURES 

2.1  Test  Sample  of  Population 

In  the  learning  analysis  in 
Block  II. 1,  analysis  of  each  learning 
objective  was  continued  to  the  point 
that  it  was  assumed  the  intended  learners 
had  already,  without  training,  mastered 
the  objective.  In  Block  II. 2,  test  items 
were  prepared  for  each  of  these  objectives.  The  first  step  in  this  block 
is  the  use  of  these  test  items  to  test  a representative  sample  of  individuals 
from  the  target  population  ot  students.  The  test  should  be  given  to  a 
large  enough  group  to  have  confidence  in  the  results.  A group  of  25-30 
students  should  be  sufficient,  particularly  if  they  are  randomly  selected 
from  the  target  population.  The  test  should  be  given  ai.d  the  results 
recorded. 

A portion  of  the  output  of  Block  11,1  might  look  like  i-igure  II. 7. 

For  simplicity,  only  the  action  portion  of  the  learning  objectives  is 
shown.  In  Block  II. 1,  the  objectives  above  the  dotted  line  were  assumed 
to  be  taught  and  those  below  the  line  were  ones  the  students  could  alreaciy 
master. 

After  testing  a sample  group  of  students,  the  test  results  might  look 
like: 


Identify  symbols 

pass 

Match  symbols  on  map  to 

legend 

fall 

Match  symbols  on  map  to 
terrain  features 

actual 

fail 

Use  legend 


fail 


Use  grid  system 
Interpret  map 
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fall 


fall 

If  so,  the  original  assumptions  were  correct,  and  the  learr.ing  objectives 
and  corresponding  test  iterj  do  not  require  modification.  In  this  caso, 
the  original  plan  wculd  be  follov^ed  and  “Identify  symbols  ' would  be 
excluded  from  the  list  of  learning  objectives  fot  the  course. 


FIGURE  II. 7:  Example  of  Part  of  A Learning  Analysis 


K the  results  were: 

Identify  symbols  fail 

Match  symbols  on  map  to  legend  fail 

Match  symbols  on  map  to  actual 
terrain  features 


fail 


fall 


U$e  legend 
Use  grid  system  fan 

Interpret  map  fan 

then  the  entry  level  of  the  students  was  overestimated  and  the  learning 
analysis  was  not  carried  far  enough.  Now  the  analysis  must  be  continued 

by  asking  the  question  "What  does  the  student  have  to  be  able  to  do  to 
master  this  objective?" 


Identify 

Symbols 


Probably  the  analysis  would  result  in: 


Identify 

Symbols 

IZZ 

I Ide  itlly 

I Colors 

1 

I Identify 

1 Shapes 

When  the  additional  learning  analysis  is  continued  to  the  point  that 
the  students  can  already  master  the  lowest  level  of  objectives,  give  the 
test  again  to  a new  group  of  students. 

If  the  results  were: 


Identify  symbols 

Match  symbols  on  map  to  legend 

Match  symbols  on  map  to  actual 
terrain  features 


pass 

pass 

pass 
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Use  legend 

pass 

Use  grid  system 

pass 

Interpret  map 

pass 

then  the  students'  skills  were  underestimated.  None  of  these  objectives 
should  be  Included  In  the  learning  objectives.  While  you  will  not  have 
to  reanalyze  objectives  or  give  additional  tests,  you  will  need  to  modify 
thn  original  plans  by  deleting  these  learning  objectives  and  their  corre- 
sponding test  items. 

A more  likely  test  result  for  30  student  Is: 


H-^ntify  symbols 

Pass 

20 

Fail 

10 

Match  symbols  on  map  to  legend 

15 

15 

Match  symbols  on  map  to  actual 
terrain  features 

15 

15 

Use  legend 

10 

20 

Use  grid  system 

6 

24 

Interpret  map 

1 

29 

Now  the  problem  Is  more  complicated.  Most  students  can  "Identify 
symbols"  but  33%  cannot.  Should  you  include  it  and  waste  the  time  of 
67%  of  the  students  or  not  include  it  and  lose  the  33%?  Neither,  if 
possible.  A better  plan  is  to  include  the  learning  objective,  but  plan 
for  those  who  can  already  master  it  to  skip  ahead  to  what  they  cannot 
do,  and  have  training  for  this  objective  available  for  the  3.3%  who 
need  it.  The  same  is  true  for  the  objectives  "2,"  "3,"  and  "4."  But 
for  objective  "5,"  determine  whether  to  have  the  six  students  who  can 
already  "use  grid  system"  go  through  the  whole  lesson.  It  Is  usually 
best  not  to  use  training  resources  and  time  to  train  people  on  what 
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they  alreacty  know.  However,  Identifying  a small  ;>erceitagp  of  unusually 
proficient  Individuals  and  developing  training  that  wM > exclude  them 
from  a few  activities  may  cost  more  than  the  potential  savings. 

Objective  "b"  certainly  fits  this  latter  category.  It  should  ba  trained 
to  all  students. 

Another  way  of  solving  the  problem  of  differences  in  the  students  Is 
to  design  the  instruction  for  most  of  the  students,  but  also  design  some 
preparatory  or  remedial  instruction  for  those  who  do  not  have  some  of  the 
entry  skills.  In  the  example,  one  would  not  Include  "Identify  symbols" 
in  the  lesson,  but  would  make  a remedial  lesson  for  such  learning  objec- 
tives that  only  a few  needed.  This  plan  may  appear  Identical  to  the  pre- 
ceding one  which  permitted  unusually  proficic.it  indlviJua'is  to  skip  certain 
objectives.  However,  there  are  ,'iome  important  differences  tiiat  will  be 
described  in  other  sections  in  this  block. 

2.2  Revise  Learning  Objectives  and  Tests  if  Required 

If  testing  of  a sample  of  the 
student  population  shows  some  of  the 
original  entry  behavior  assumptions 
to  be  wrong,  go  back  to  Blocks  II.l 
and  11.2,  make  the  necessary  revisions 
to  the  learning  objectives  and  tests, 
and  if  new  objectives  were  added,  try 
out  the  revised  product  on  a new  group 

of  students.  This  revise/test  procedure  must  be  repeated  until  there  is 
a good  match  between  learning  objectives  and  the  entry  behavior  of  the 
prospective  students. 
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2.3  Detenwlne  Heed  for  an  Entry  Test 
An  entry  test  Is  not  required 
If  the  external  entry  requirements 
of  the  system  equal  or  exceed  the 
entry  requirements  of  the  course. 

EXAHPLE 


After  making  the  final  revision  to  the  learning  objectives 
and  testing  these  on  a group  of  30  students,  a p .'tion  of 
the  final  results  might  look  like  this:  (This  Is  a 
different  example  from  the  one  used  earlier.) 


Learning  Objective 

Pass 

F- 

1 

30 

0 

2 

30 

0 

3 

30 

0 

4 

22 

8 

5 

19 

11 

6 

14 

16 

7 

8 

22 

8 

5 

25 

9 

1 

29 

10 

2 

28 

If  the  entry  behavior  assumptions  were  that  entering  students 
mastered  the  first  three  objectives  but  not  the  others,  you 
now  will  have  test  data  to  show  that  those  assumptions  were 
correct.  Furthermore,  If  analysis  of  the  external  require- 
ments (external  requirements  were  discussed  In  the  Introduction 
to  this  block)  indicate  that  all  entering  students  are  likely 
to  be  able  to  master  the  first  three  objectives,  there  is  no 
need  for  an  entry  test  so  long  as  the  external  requirements 
remain  unchanged.  If  the  first  learning  objective  above 
requires  reading  at  the  fourth  grade  level,  and  the  personnel 
system  requires  a high  school  level  of  education  for  entiy  into 
the  DOS,  an  entry  test  to  check  reading  level  is  probably  not 
needed . 


71 


More  likely  the  external  entry  requ1ref>ents.  which  are  usually  designed 
only  to  be  general  personnel  screening  Instruments,  will  be  too  general  to 
Indicate  whether  students  wHl  meet  the  entry  requirements  for  specific 
Instruction.  This  Is  particularly  true  of  advanced  technical  courses. 

More  likely  you  will  need  an  entry  test  and  will  have  to  make  decisions 
as  to  what  action  to  take  with  entering  students  who  fall  parts  of  the  cest. 


EXAMPLE 


After  making  the  final  revisions  to  your  learning  objectives 
and  testing  these  on  a group  of  30  students,  the  results  are 
more  likely  to  look  like: 


Learning  Objective 

Pass 

Fai 

1 

28 

2 

2 

27 

3 

3 

29 

1 

4 

25 

5 

5 

20 

10 

6 

15 

15 

7 

12 

IB 

8 

5 

25 

9 

0 

30 

10 

0 

30 

Assuming  that  the  course  will  include  the  last  six  objectives 
but  not  include  the  first  four  objectives,  an  entry  test  will 
be  required  for  use  in  identifying  students  who  do  not  have 
the  required  entry  behaviors,  and  in  identifying  the  particu- 
lar areas  where  individual  students  fail  to  meet  the  entry 
requirements. 


In  the  above  example,  a decision  also  must  be  made  as  to  what  action 
to  take  with  students  who  fail  to  meet  the  entry  requirements  for  the  course. 
The  alternatives  are: 

1.  Suggest  that  the  personnel  section  change  the  external 
requirements  for  entry  into  the  DOS.  There  are  several 
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disadvantages  to  this.  The  suggestion  may  be  denied. 

Even  if  it  is  not  denied,  much  time  and  effort  probably 
will  be  required  before  the  change  will  be  made- -time 
and  effort  that  might  be  spent  more  profitably  developing 
better  instruction.  And  finally,  such  suggestions  tend  to 
present  a negative  picture  of  the  capabilities  of  the 
training  establishment. 

2.  Assume  from  the  beginning  that  some  students  will  not 
pass  the  course.  This  has  the  disadvantage  o-f  wasting 
training  resources  and  student's  time;  however,  it  may 
be  the  only  alternative  if  time  for  training  is  severely 
limited.  These  decisions  must  be  made  by  managers  aware 
of  the  total  training  problem. 

3.  Prepare  remedial  lessons  to  correct  the  student's  defi- 
ciencies before  they  begin  the  course.  While  this  will 
require  additional  training  resources,  it  usually  is  the 
best  solution. 

2.4  Develop  Entry  Test 

Details  of  test  development  are  given 
In  Block  1 1. 2.  Since  test  items  have  already 
been  developed  for  each  learning  objective, 
the  effort  here  is  primarily  one  of  selecting 
and  organizing  the  test  items.  Generally, 
only  the  objectives  that  the  intended  stu- 
dents must  have  mastered  in  order  to  begin 
the  course  are  included  in  the  entry  test. 


] 

I 

•1 

1 

I 


I 

i 


1 

i 

j 
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2.S  Detennine  Requirement  for  a Pretest 
While  you  probably  will  develop 
some  kind  of  pretest  for  use  in  obtaining 
more  meaningful  internal  evaluation  data 
in  Block  V.l,  you  may  or  may  not  require 
a pretest  as  a permanent  part  of  the  course. 

Other  than  for  internal  evaluation  purposes 
during  the  development/evaluation  stages  of 
the  course,  there  are  two  reasons  for  using 
a pretest.  These  are: 

1.  To  determine  if  entry  behavior  of  students  has  changed 
over  time.  While  the  entry  test  discussed  earlier  will 
reveal  a decrease  in  incoming  students'  abilities,  it 
ordinarily  cannot  reveal  an  increase  in  abilities.  Since 
the  pretest  tests  the  entire  course,  it  will  show  up  any 
increases  in  the  number  of  incoming  students  who  have 
already  mastered  parts  of  the  instruction.  If  this  were 
the  only  reason  for  giving  the  pretest,  and  if  the  test 
was  lengthy,  a good  decision  might  be  to  use  the  pretest 
periodically  rather  than  for  all  incoming  students. 

2,  To  determine  which  parts  of  the  instruction  certain  students 
have  already  mastered,  so  as  to  assign  individuals  to  only 
those  parts  they  reed  to  learn.  This  can  result  in  a 
considerable  savings  in  instructional  resources  and  students' 
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i:;] 

(■ 


.r 


74 


This  type  of  trainee  assignment  is  difficult  to  achieve  when  Instruc- 
tional flexibility  Is  limited.  Many  courses  are  offered  only  In  the  group 
block  schedule  ende.  Here  all  students  are  exposed  to  the  same  Instruction 
at  the  same  tine.  One  way  that  pretest  can  be  used  In  such  situations  Is 
Illustrated  In  Figure  11.8  below. 


TIME 

Weak  Week 


This  course  Is  five  weeks  long.  The  pretest  is  divided  Into  five 
subsets,  each  Including  the  objectives  from  a single  week  of  Instruction. 
All  students  will  take  the  pretest  and  exempt  any  week  of  instruction 
that  their  scores  indicate  they  have  already  mastered.  A student  who 
obtains  a passing  score  on  subsets  "1,"  "2,"  and  "3,"  might  begin  course- 
work  with  the  group  currently  in  the  fourth  week.  A student  who  passes  the 
entire  test  mey  exempt  the  whole  course. 
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There  are  no  fixed  rules  for  when  to  use  a pretest.  If  a large 
majority  of  students  fail  all  Items  on  the  pretest,  the  potential 
savings  In  not  requiring  students  to  restudy  what  they  already  know 
probably  Is  less  than  the  cost  of  testing.  In  such  a case,  a pretest 
should  not  be  used  as  a '"egular  part  of  the  course.  Occasionally  pre 
testing  is  itn^ractical  because  of  resource  constraints  where  special 
facilities  are  required.  If  there  is  doubt  as  to  the  advisability  of 
using  a pretest,  the  best  <teci.sion  probably  is  to  use  it,  at  least 
until  experience  shows  that  it  serves  no  useful  purpose. 

2.6  Develop  Pretest 

The  pretest  generally  is  either 
Identical  to  the  posttest  developed 
In  Block  II. 2 or  an  alternate  version 
of  that  test.  If  the  posttest  is 
unusually  lengthy  or  requires  use 
of  scarce  resources,  the  pretest 
may  be  a less  detailed  version  of 
the  posttest.  Details  of  test 
development  are  given  in  Block  II. 2. 

The  effort  in  this  block  is  primarily 
one  of  selecting  and  organizing  the 
test  items,  and  determining  what 
action  will  be  taken  when  specific 
items  on  the  pretest  are  passed  by 
students  taking  the  test. 
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3.Q  Outputs 

The  outputs  of  this  block  should  consist  of: 

3.1  Products 

1.  teaming  objectives  and  tests  revised  to  match  entry 
behavior  of  students. 

2.  Entry  test  if  required. 

3.  Pretest  if  required. 

3.2  Other  Doc ufnen  :ati on 

1.  Summery  statement  of  testing  activities  and  changes 
made  as  a result  of  testing. 

2.  Rationale  for  decision  on  whether  to  use  an  entry  test. 

3.  Rationale  for  decision  on  whether  to  use  a pretest. 


EXAMPLE 


The  entry  behaviors  for  the  "remove  main  drive 
shaft  from  OH-58  helicopter"  learning  objective 
described  In  Block  II. 1 (page  3b)  Include; 


a.  Identifying  parts 

b.  Identifying  tools 

c.  selecting  correct  tools  for  specific  purposes 

d.  using  tools 

e.  reading  procedures 

f.  using  diagrams  and  schematics 

A sample  test  Item  would  be: 

Identify  the  numbered  parts  in  the  diagram; 


INPUTS 


I INPUTS 


completed  learning 

ANALYSIS  AND  OBJECTIVES 
FROM  BLOCK  111  AND  TEST 
ITEMS  FROM  BLOCK  11.2 


EXTERNALLY 

CONTROLLED 

ENTRY 

REQUIREMENTS 


f 

P 

i: 

I 


DEVELOP 

ENTRY 

I'Esr 

2.4 


DETERMINE 
REQUIREMENT 
FOB  A 
PRETEST 


FIGURE]!. 6:  Fio'.rcharl  for  Block  IT.3,  DESCRIBE  ENTRY  BEHAVIOR. 


DETERMINE  SEQUENCE  AND  STRUCTURE 


I.O  INTRODUCTION 


The  term  "learning  objective"  Is  used  In  this  block  to  refer  to 
terminal  leaniing  objectives,  learning  objectives,  and  learning  steps. 

In  this  block,  you  will  arrange  learning  objectives  in  the  sequence 
in  which  instruction  will  be  presented  to  the  student.  In  addition, 
you  will  arrange,  or  structure,  related  learning  objectives  into  major 
groups  to  simplify  further  handling  of  broader  content  areas. 

The  purpose  of  sequencing  learning  objectives  is  to  help  assure 
that  when  instructional  materials  are  developed  learning  each  objective 
is  placed  in  optimum  relationship  to  other  learning  objectives.  Proper 
sequencing  will  help  produce  the  most  learning  in  the  shortest  period 
of  time,  will  help  the  learner  make  the  transition  from  one  skill  or 
body  of  knowledge  to  another,  and  will  assure  that  the  supporting 
knowledge  and  skills  are  acquired  before  dependent  subject  matter  is 
introduced.  The  best  sequence  is  the  one  that  works  best  for  the  student. 
While  it  obviously  would  be  impossible  to  evaluate  every  possible  sequence, 
there  are  alternatives  to  consider. 

In  Block  1.2,  tasks  were  selected  for  training.  In  Block  II. 1, 
terminal  learning  objectives  were  derived  for  each  of  these  casks. 

When  the  infonnation  and  skills  required  to  master  each  terminal  learning 
objective  were  considered,  a number  of  learning  objectives  and  learning 
steps  generally  were  added.  This  was  necessary  since  the  terminal 
learning  objective  was  too  complex  for  a first  step  in  the  learning 
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process  for  most  of  your  students.  The  terminal  learning  objectives 
were  broken  down  Into  less  complex  learning  objectives  until  the 
assumed  entry  level  of  the  students  was  reached. 

Next,  In  Block  II. 3,  these  original  assumptions  about  the  entry 
behaviors  of  students  were  tested.  If  the  assumptions  were  wrong, 
procedures  outlined  in  Block  II.l  were  used  to  add  or  delete  learning 
objectives  until  the  lowest  level  of  learning  objective  matched  the 
entry  behaviors  of  the  students.  This  revised  group  of  learning  objec- 
tives is  what  you  need  to  begin  work  in  this  block.  If  each  learning 
objective,  along  with  its  conditions,  standards,  and  test  items  where 
applicable,  is  listed  on  a separate  form  such  as  the  Learning  Objec- 
tive Mfslys-;'.  Worksheet  shown  in  Block  II.l,  the  actual  sorting, 
sequencing,  and  grouping  will  be  simplified. 

The  steps  in  determining  sequence  and  structure  are  shown  in 
Figure  II. 9,  the  fold-out  page  at  the  end  of  this  block. 


2.0  PROCEDURFS 


2.1  Determine  Relationships  Between  Learning  Ob.iertives 
In  order  to  sequence  two  learning 
objectives,  you  first  must  determine 
the  relationship  between  them.  Two 
learning  objectives  may  have: 

1.  a dependent  relationship. 


in  that  mastery  of  one  requires 
prior  mastery  of  the  other; 
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2.  an  Independent  relationship,  in  that  they  are  totally 
unrelated  and  independent  of  each  other;  or 

3.  a supportive  relationship,  in  that  some  transfer  of  learning 
takes  place  from  one  learning  objective  to  the  other. 

Table  II. 7 contrasts  these  relationships,  gives  examples  of  each, 
and  shows  how  the  relationship  affects  sequencing. 

The  table  might  give  the  impression  that  all  pairs  of  learning 
objectives  that  do  not  have  a dependent  relationship  have  either  a 
completely  independent  or  completely  supportive  relationship.  But, 
most  pairs  of  learning  objectives  that  do  not  have  a dependent  rela- 
tionship probably  fall  somewhere  between  these  two  extreme  positions. 
Between  most  pairs  of  learning  objectives,  some  transfer  of  learning 
may  take  place.  This  makes  for  some  difficulty  in  sequencing  these 
learning  objectives.  If  you  are  convinced  that  very  little  or  no 
transfer  of  learning  is  likely  to  take  place  between  bvo  learning 
objectives,  conside*'  the  relationship  an  independent  one.  When  in 
doubt,  consider  the  relationship  a supportive  one.  Guidelines  for 
sequencing  learning  objectives  with  independent  and  supportive  rela- 
tionships will  be  presented  later  in  this  block. 


2.2  Sequence  Learning  Objectives  with  Dependent  Relationships 

Sequencing  learning  objectives  with  [ 

dependent  relationships  usually  is  a 
simple  step,  particularly  when  the 
learning  objectives  were  arranged  in 
the  proper  hierarchy  in  Block  II. 1. 
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TABLE  H.7  Types  of  Relationships  Between  Learning  Objectives 


DEPENOEKT 


INDEPENDENT 


SUPPORTIVE 


Skills  and  knowledges 
1n  one  learning  objec- 
tive are  closely 
related  to  those  in 
the  other  learning 
objective. 


Skills  and  knowledges 
in  one  learning  objec- 
tive are  unrelated  to 
those  in  the  ether 
learning  objective. 


Skills  and  knowledges 
in  one  learning  objec- 
tive have  some  rela- 
tionship to  those  in 
the  other  learning 
objective. 


To  ntuster  one  of  the 
learning  objectives., 
it  is  first  necessary 
to  master  the  other. 


Mastering  one  of  the 
learning  objectives 
does  not  simplify 
mastering  the  other. 


The  learning  Involved 
in  mastery  of  ore 
learning  objective 
transfers  to  the 
other,  making  learn- 
ing Involved  in  the 
mastery  of  the  other 
easier. 


EXAMPLES: 

In  math,  in  order  to 
learn  multiplication 
one  must  first  learn 
addition. 

One  cannot  send 
messages  in  Morse 
Code  without  first 
having  nwstered  the 
codes  for  each  of 
the  letters  and 
numbers.  The  "sending" 
skills  are  totally 
dependent  on  the 
prior  learning. 


Examples: 

For  a yeoman,  "type 
letters  from  drafts" 
is  independent  of 
"maintain  files." 

For  a wheeled  vehicle 
mechanic,  "adjust 
carburetor"  is  inde- 
pendent of  "torgue 
engine  head  studs." 

Ip  both  examples, 
knowing  how  to  do 
one  would  not  help 
much  with  the  other. 


In  general,  the 
learning  objectives 
can  be  arranged  in 
any  sequence  wi th- 
out  loss  of  learning. 


Examples: 

"Assemble  weapon"  has 
a supportive  relation- 
ship to  "disassertble 
weapon." 

"Drive  a U con  truck" 
has  a supportive 
relationship  to  "drive 
a 2*5  ton  vehicle." 

In  both  examples, 
learning  to  do  one 
would  help  consider- 
ably in  learning  to 
do  the  other. 


The  learning  objectives 
should  be  placed  close 
together  in  the  sequent 
to  pennit  optimum 
tronst'er  of  learning 
from  one  learning  objec 
ti  ve  to  the  other . 
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For  example,  a terminal  learning  objective  with  its  supporting  learning 
objectives  and  learning  steps  might  look  like  Figure  11.10.  You  will 
note  that  for  convenience,  only  the  action  components  of  the  objective 
are  listed.  Before  you  can  master  the  terminal  learning  objective  of 
"get  from  point  A to  point  B,"  you  must  be  able  to  do  two  learning 
objectives:  "orient  map  and  con^Jass,"  and  "hike  with  a pack."  Some  of 

the  learning  objectives  under  "orient  map  and  compass"  have  dependent 
relationships.  These  hierarchally-ordered  learning  objectives  are, 
therefore,  easy  to  sequence.  One  sequence  is; 

1.  Identify  symbols 

2.  a.  Match  symbols  to  actual  terrain 

b.  Match  symbols  to  legend 

3.  a.  Use  legend 

b.  Use  grid  system 

4.  Interpret  map 

Some  of  the  learning  objectives  present 
more  of  a sequencing  problem,  however.  For 
example,  in  Figure  II. 10,  which  should  be 
placed  first,  "read  compass  values  from 
scale"  or  "align  points  with  scale?"  These 
can  be  placed  in  either  order,  so  long  as 
they  are  both  placed  before  "read  compass," 
and  as  long  as  "read  compass  from  scale" 
comes  after  the  five  learning  objectives 
under  it. 
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FIGURE  H .10:  Learr.ing  Analysis 


However,  If  you  place  "align  points  with 
scale"  within  learning  obifTtlve  1 through  6 


8 


you  would  interfere  with  -he  dependent 
sequence  of  those  learning  objectives. 
Therefore,  unless  you  had  a good  reason 
for  doing  otherwise,  you  would  place 
"align  points"  either  after  "read  compass 
values  from  scale"  or  before  the  group 
of  learning  objectives  that  have  a 
dependent  relationship  to  “reafl  compass 
values  from  scale. " 


2 


1 3 4 


OR 


2 4 5 


There  are  other  independent  relationships  in  Figure  II. 10.  For 
example,  "sight  compass"  has  a dependent  relationship  with  "use  compass 
But,  since  it  has  an  independent  relationship  with  the  other  learning 
objectives,  "sight  compass"  can  be  placed  anywhere  in  the  sequence  so 
long  as  it  precedes  "orient  map  and  compass."  Again,  as  with  "align 
points  with  scale,"  unless  you  had  a good  reason  for  doing  otherwise, 
you  would  avoid  placing  "sight  compass"  within  a group  of  learning 
objectives  that  have  dependent  relationships  with  each  other. 
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Wiile  thf  abrve  are  examples  of  learning  objectives  with  dependent 
relationships  that  support  a terminal  learning  objective,  the  same  rules 
apply  to  sequencing  tenninal  learning  objectives.  For  example,  one 
terminal  learning  objective  might  be  to  operate  a certain  piece  of  equip- 
ment, while  another  terminal  learning  objective  might  be  to  instruct  others 
in  the  proper  operation  of  the  eouipment.  Most  likely  a dependent  rela- 
tionship exists  between  these  two  terminal  learning  objectives;  therefore, 
you  would  place  "operate  equipment"  before  "Instruct  others." 

Generally,  when  you  place  a particular  terminal  learning  objective  In 
a certain  position  in  the  sequence,  you  will  locate  the  learning  objec- 
tives that  support  the  terriinal  learning  objective  with  it.  This  is 
because  of  the  dependent  relationship  of  the  learning  objective  to  the 
terminal  learning  objective,  and  because  of  the  transfer  of  learning 
that  is  more  likely  to  occur  when  closely  related  learning  objectives 
are  kept  together.  Some  exceptions  to  this  rule  will  be  pointed  out 
later  in  Section  2.5  of  this  block. 

2.3  Sequence  Learning  Objectives  with  Supportive  Relationships 

The  one  basic  rule  for  sequencing 
learning  objectives  with  supportive 
relationships  was  just  stated  in  the 
previous  section.  It  is  to  place  the 
learning  objective  as  close  together 
in  the  sequence  as  practical  so  that 
optimum  transfer  of  learning  can  take 
place. 

In  addition  to  this  basic  rule,  there  are  several  other  good  reasons 
for  placing  learning  objectives  close  together  in  the  sequence.  You  may 
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wish  to  place  learning  objectives  close  together  If  the  conditions  under 
which  the  learning  objectives  are  carried  out  are  Identical  or  similar, 
and  If  the  conditions  are  difficult  or  expensive  to  produce  at  random 
times.  For  example,  If  several  lear.iing  objectives  have  conditions  such 
as  "at  night,"  "on  muddy  terrain,"  or  "when  flying  at  an  altitude  of 
20,000  feet,"  you  will  probably  wish  to  place  the  "at  night"  learning 
objectives  together,  the  "on  mud4y  terrain"  learning  objectives  together,  etc. 

Also,  if  a particular  piece  of  equipment  must  be  available  In  order 
to  accomplish  a group  of  learning  objectives,  and  1f  you  are  not  Hi  .ly 
to  have  continuous  access  to  that  equipment,  you  will  probably  wish  to 
group  the  learning  objectives  that  use  that  particular  equipment.  In 
addition,  you  will  want  to  keep  the  order  of  the  group  of  learning 
objectives  within  the  total  program  as  flexible  as  possible.  For 
example,  if  the  learning  objective  is  "perform  maintenance  on  helicopter," 
you  will  have  to  wait  until  the  helicopter  is  available  for  maintenance 
before  you  can  accomplish  the  objective. 

A remaining  question  is;  Which  of  two  supportive  learning  objectives 
should  be  placed  first  in  the  sequence?  At  present,  there  are  no  hard 
.'i.'iJ  fa;  ‘ rules  for  sequencing  objectives  that  have  supportive  relation- 
ships. 3ince  there  are  no  rules,  do  not  specify  sequence  at  this  point. 

The  iiist;uctors  and  materials  developers  can  select  tlie  sequence  in  Blo.-k 
III. 4,  where  logic-based  rather  than  data-based  sequencing  patterns  like 
simple  to  complex,  chronological  order,  etc.  are  presented. 

While  the  effectiveness  of  various  sequencing  techniques  is  still 
under  question,  a number  of  conclusions  have  been  reached  about  sequenc- 
ing in  general. 
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1.  Sequencing  effects  are  long-range.  The  advantages  or  dis- 
advantages of  using  any  sequencing  scheme  will  not  likoly 
show  up  Imnsdiately . Therefore,  end-uf-course  tests  should 

be  used  for  evaluating  effectiveness  of  sequencing  techniques. 
Within-course  tests  of  small  portions  of  the  course  are  not 
likely  to  reveal  the  true  effects  of  sequence. 

2.  Sequence  is  Important  to  low  aptitude  students.  Students 
who  have  a high  aptitude  for  the  subject  matter  will  learn 
it  in  spite  of  sequencing.  The  lower  aptitude  of  the 
learner  for  the  content,  the  more  important  it  becomes 
that  sotre  type  of  sequence  and  structure  is  provided. 

i.  Sequence  is  important  with  unfamiliar  materials.  Students 
who  are  familiar  with  materials  will  learn  regardless  of  order 
of  presentation.  But  as  material  becomes  increasingly  Unfamil 
iar  to  the  student,  the  importance  of  sequence  increases. 

4 . Sequence  is  important  to  non-redunJant  materials.  Some 
instructional  material''  are  especially  redutidant , stating 
important  points  over  and  over  again.  Seq'uencing  is  not 
especially  important  with  these  ma ; ei-i a 1 s , because  the 
student  can  pick  up  the  second  time  anything  he  has  missed 
the  first  time.  But  if  materials  are  non-redundant  and 
state  their  points  only  once,  it  is  important  that  the 
materials  be  sequenced  according  to  some  rationale. 
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2.4  Sequence  Learning  Objective;,  with  Independent  Relationships 
As  was  mentioned  earlier,  learning  objectives 
with  independent  relationships  may  be  arranged 
in  any  sequence;  however,  such  learning  objec- 
tives generally  will  not  be  placed  between 
learning  objectives  chat  have  a dependent 
relationship  to  each  other.  Any  of  the  guide- 
lines listed  for  sequencing  learning  with  a 
supportive  relationship  may  also  be  used 
for  sequencing  objectives  with 
independent  relationships.  However,  with  the  latter,  you  need  not  be 
concerned  about  locating  the  learning  objectives  close  together  in  the 
sequence  unless  condition  or  equipment  constraints  indicate  otherwise. 


2.5  Sequence  Comiiion-ractor  Learning  Objectives 

Comtnon- factor  learning  objectives  are 
learning  objectives  that  art  identical  or 
learning  objectives  that  have  identical 
action  words  and  similar  objects  of  the 
action  in  the  learning  objective  statement. 

Figure  II.  11  contains  an  example  of  the 
former  and  Figure  11.12  contains  an 
example  of  the  latter. 

Consider  the  learning  objective  "dosolder."  It 
"remove  defective  capacitor"  and  "remove  transfunner 


occurs  under  both 
. " !iu  t you  wi  1 1 


want  to  teach  desolderiiig  only  one  time,  'iherefore,  yoi-  have  two  choices. 
You  can  delete  the  learning  objective  in  all  cases  except  the  first  time 
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FSGUREIE.1 1:  Example  of  Partial  Learning  Analysis  Showing  Common  Element  Learning  Objectives 


FIGUREIE.12;  Example  of  Partial  Learning  Analysis  Showing  Similar  Learning  Objectives 


Best  Available  Cqd^/ 
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it  appears  In  yout'  list  of  sequenced  objectives.  Or,  since  desoldering 
is  a prerequisite  to  a number  of  other  objectives,  you  can  delete  it  from 
every  place  it  appears  and  teach  it  (probably  along  with  soldering,  if 
soldering  is  required  in  the  course), near  the  beginning  of  the  course. 

In  Figure  11.12,  note  that  "identify  capacitors,"  "Identify  transistors, 
and  "identify  resistors"  are  required  at  different  points.  One  logical 
way  to  handle  these  comnon  factor  learning  objectives  is  to  group  all 
"identify"  learning  objectives  and  place  them  near  the  beginning  of  the 
course. 

While  common-element  learning  objectives  generally  should  be  listed 
early  in  the  sequence  because  they  represent  information  or  skills  basic 
to  many  learning  objectives,  the  delay  between  introducing  such  objectives 
and  the  actual  application  of  the  objectives  should  be  minimized.  This 
is  so  that  material  learned  at  the  beginning  of  training  will  not  be  for- 
gotten by  the  time  it  is  practiced  as  a part  of  other  learning  objectives. 

2.6  Structure  Learning  Objectives  into  Groups 

While  sequencing  learning  objectives, 
you  undoubtedly  realized  that  you  were 
dealing  with  quite  a few  items;  so  many, 
in  fact,  that  you  might  have  had  difficulty 
in  getting  a clear  picture  of  the  relation- 
ship between  all  of  them.  You  can  partially 
overcome  this  difficulty  by  dividing  the 

learning  objectives  into  Icrge  groups  that  will  provide  organized, 
manageable  blocks  of  content  with  which  you  can  work.  How  many  learning 
objectives  you  group  into  a block  is  an  arbitrary  matter.  At  this  point. 
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you  are  simply  Identifying  learning  objectives  that  can  be  grouped  because 
of  the  close  relationship  between  the  learning  objectives  Involved.  Group- 
ing can  help  you  In  the  following  ways: 

A.  Grouping  content  areas  early  will  help  you  get  a clearer 
picture  of  the  scope  and  nature  of  your  developing  program. 

B.  Grouped  learning  objectives  can  be  worked  on  Independently. 

If  one  person  1s  responsible  for  development  of  the  total 
system,  he  can  work  on  later  groups  If  all  the  material  Is 
not  available  for  earlier  groups. 

C.  Mien  more  than  one  person  Is  assigned  to  develop  course 
materials,  different  people  can  be  assigned  responsibility 

for  different  groups.  Each  person  then  can  work  relatively  ' 
Independently  on  a group  of  learning  objectives,  conferring 
with  others  as  needed. 

Following  are  some  guidelines  that  should  be  helpful  In  dividing 
the  learning  objectives  Into  groups: 

A.  Remember  that  some  grouping  of  tasks  was  done  In  Block  1.5. 
While  this  was  a tentative  grouping,  the  learning  objectives 
for  these  groups  of  tasks  should  be  kept  together  unless  a 
decision  has  been  made  to  change  the  grouping. 

B.  For  a particular  group,  select  learning  objectives  that  bear 
a close  relationship  to  each  other.  Combined,  they  should 
make  a self-contained  group. 

C.  Coirblne  learning  objectives  so  that  the  group  has  a natural 
beginning  and  ending  point. 

D.  Be  sure  all  learning  objectives  are  Included  somewhere. 
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F..  In  general,  learning  objectives  within  your  groups  will  remain 
In  exactly  the  same  order  as  your  original  sequence.  However, 
since  dividing  learning  objectives  Into  groups  should  help 
clarify  your  total  program,  do  not  hesitate  to  re-sequence 
objectives  where  necessary. 

F.  Oo  not  assume  that  your  group  of  learning  objectives  Is 
unalterable.  Subsequent  decisions  mqy  require  you  to  re- 
evaluate what  should  constitute  a group. 

3,0  OUTPUTS 

The  outputs  of  this  block  should  consist  of: 

3.1  Products 

The  final  list  of  sequenced  and  structured  learning  objectives. 

(See  examples  that  follow.) 

3.2  Other  Documentation 

1.  Rationale  for  any  major  sequencing  decisions 

2,  Rationale  for  any  major  changes  made  In  the  sequence  as  a 
result  of  Phase  III,  IV,  and  V activities 

EXAMPLES 

Structured  learning  objectives  for  the  Job:  OH-58  Helicopter 

Repainnan  (Action  portion  only) 

1.  Main  Transmission  Functions 

A.  Main  transmission  pylon  supports 

1.  inspect  main  transmission  pylon  supports 

2.  install  main  transmission  pylon  supports 

3.  remove  main  transmission  pylon  supports 


B.  Main  Transmission  drag  pin  assembly 

1.  procure  main  transmission  drag  pin  assembly 

2.  Inspect  main  transmission  drag  pin  assenbly 

3.  install  main  transmission  drag  pin  assembly 

4.  remove  main  transmission  drag  pin  assembly 

C.  Main  Transmission  Drive  Quill  Seals 

1.  inspect  main  transmission  drive  quill  seals 

D.  Main  Transmission  Thermo  Switch 

1.  procure  main  transmission  thermo  switch 

2.  inspect  main  transmission  thermo  switch 

3.  install  main  transmission  thermo  switch 

4.  remove  main  transmission  thermo  switch 

E.  Main  Transmission  Oil  Filter  Head  Assembly 

1.  inspect  main  transmission  oil  filter  head  assembly 

2.  install  main  transmission  oil  filter  head  assenbly 

3.  remove  main  transmission  oil  filter  head  asseml'ly 


Sequenced  learning  objectives  for  the  task:  Maintain  power 
train  system  on  OH-58  helicopter.  (After  the  task  analysis 
in  Block  II. 1.  Figure  II. 3.) 

Point  to  and  Name: 

1.  each  component  of  an  actual  OH-58  helicopter  drive 
shaft  while  referring  to  an  exploded  view  diagram 
of  the  drive  shaft. 

2.  Disassemble  the  OH-58  helicopter  drive  shaft  while 
referring  to  a checklist  which  names  but  does  not 
picture  the  components. 

3.  Dissassemble  the  drive  shaft  without  the  aid  of  a 
checklist  or  drawing. 

4.  Assemble  a drive  shaft  while  referring  to  a check- 
list which  names  but  does  not  picture  the  con^)onents. 

5.  Assemble  the  drive  shaft  without  the  aid  of  a check- 
list or  drawing. 
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tests . Reston,  Va. : Applied  Science  Associates,  "1974.” 


Criterion-referenced  tests  measure  what  an  individual  can  do  or 
knows  compared  to  what  he  must  be  able  to  do  or  know  in  order  to 
successfully  perform  a task.  This  manual  provides  guidance  on  the 
construction  and  use  of  criterion-referenced  tests. 


Thorndike,  E.  L.  (Ed.).  Educational  measurement.  Washington:  American 

Council  on  Education,  1971. 

This  text  is  a collection  of  papers  dealing  with  issues  in  the  con- 
struction, design  and  administration  of  tests.  A background  dis- 
cussion of  measurement  theory  relevant  to  a clear  understanding  of 
testing  is  elaborated  upon  in  several  articles. 
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Thorndike,  E.  I..,  (Ed.).  Educational  mcasureflieiit.  Washington,  D.C. 
American  Council  of  Education,  1971 . 


Thorndike's  book  is  a collection  of  articles  concerned  with  tra- 
ditional measurement  questions.  Several  articles  deal  wicli  test 
construction  and  design,  measurement  theory,  and  analyzing  data 
from  test  items.  The  naive  test  designer  would  find  these 
articles  helpful. 
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Department  of  the  A1r  Force.  Handbook  for  designers  of  Instructional 

systems:  Planning,  developing,  and  validating  instruction  (Vo1.1V. AFP 
50-58).  Washington,  D.C.:  Headquarters,  United  States  Air  Force, 

July  1973. 


Once  the  objectives  for  a course  of  study  have  been  defined.  It  is 
necessary  to  sequence  objectives  so  that  transition  from  one  skill 
to  another  will  be  optimal.  Objectives  are  sequenced  and  structured 
to  provide  a smooth  flow  of  learning  opportunities. 


Gagne,  R.  M.,  & Briggs,  L.  J.  Principals  of  instructional  design. 
Nei/  York:  Holt,  Rinehart  and  Winston,  Inc.,  1974 


Course  planning  involves  the  sequencing  of  instruction  to  ensure 
that  relevant  prerequisite  or  subordinate  capabilities  are  learned 
before  superordinate  tasks.  One  method  of  sequencing  instruction 
ij  to  design  learning  hierarchies  which  trace  prerequisite  capabil- 
ities of  an  objective  to  the  entry  behavior  level  of  the  trainee. 
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ACRONYMS 


AFS  - Air  Force  Speciality 

AR  - Army  Regulations 

CMI  - Computer  Managed  Instruction 

CRT  - Criterion  Referenced  Test 

CODAP  - Comprehensive  Occupational  Data  Analysis  Programs 

DOS  - Defense  Occupational  Specialities 

FM  - Field  Manuals 

FOOT  - Formal  On  The  Job  Training 

GED  - General  Educational  Development 

HQ  - Headquarters 

ISD  - Instructional  Systems  Developmient 

ISS  - Installation  Support  School 

ITV  - Instructional  Television 

JPA  - Job  Performance  Aids 

JPM  - Job  Performance  Measure 

KOR  - Knowledge  of  Results 
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LO  - Learning  Objective 
LS  - Learning  Step 

MOOB  - Military  Occupational  Data  Bank 

MOS  - Military  Occupational  Specialities  (Army/Marine  Corps) 

MilO  - Modification  Work  Orders 

i 

NIH  - Not  Invented  Here 

NOTAP  - Naval  Occupational  Task  Analysis  Program 

OJT  - On  The  Job  Training 

OSR  - Occupational  Survey  Report 

POI  - Program  of  Instruction 

QQPRI  - Qualitative  and  Quantitative  Personnel  Requireiiients  Identification 

RS  - Resident  School 

SME  - Subject  Natter  Expert 

SMP  - System  Master  Plan 

SOP  - Standing  Operation  Procedures,  Standard  Operating  Procedures 
STEP  - Self-Teaching  Exportable  Package 
TAK  - Trainer  Appraisal  Kit 
TI  - Traditional  Instruction 


TLO  - Terminal  Learning  Objective 
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TM  - Technical  Manuals 

TOE  - Tables  of  Organization  and  Equipment 

TRADOC  - U.S.  Training  and  Doctrine  Command 


GLOSSARY 


ABILITY  GROUPING:  Arrangement  whereby  students  are  assigned  to  groups 
on  the  basis  of  aptitude  testing. 


ABSOLUTE  STANDARDS:  A statement  defining  the  exact  level  of  performance 
required  of  a studetst  as  a demonstration  that  he  has  mas- 
tered the  course  objectlve(s).  Criterion-referenced  tests 
are  usually  based  on  an  absolute  standard. 


ACHIEVEMENT  GROUPING:  Arrangement  whereby  students  are  assigned  to 

groups  according  to  their  performance  on  pretests  of  units 
of  the  course. 


ACTION:  Occurs  In  terminal  learning  objectives  and  learning  objectives; 

describes  the  specific  behavior  the  learner  is  to  exhibit 
after  training. 


ACTION  VERBS:  Verbs  that  convey  action  and  reflect  the  type  of  learn- 
ing that  is  to  occur.  Action  verbs  must  reflect  behaviors 
that  are  measureable,  observable,  verifiable,  and  reliable. 


ACTIVITY  STEP:  One  simple  operation  or  movement  that  comprises  part  of 
a job.  A job  performance  standard  consists  of  a list  of 
these  operations  or  movements. 


ADJUNCT  PROGRAMMING:  A method  of  combining  the  features  of  good  exist- 
ing Instructional  materials  (e.g.,  films,  textbooks)  with 
special  directions  or  questions  to  guide  the  learner. 


AtWINISTRATIVE  CRITERIA:  In  media  selection,  the  options  that  course- 

ware be  developed  locally  or  at  some  central  location. 


ALGORITHM:  A rule  or  procedure  for  accomplishing  a task  or  solving 
a problem. 


ALPHANUMERIC:  Refers  to  a combination  of  letters  and  numbers;  for 

example,  on  the  keyboard  of  a teletype. 


ALTERNATE  PATH:  Refers  to  elements  which  have  relationships  in  which 

the  specific  situation  encountered  determines  the  appro- 
priate sequence,  or  it  may  be  another  way  of  meeting  the 
same  objective. 
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ASSESSMENT:  A Judgment  of  the  effectiveness  and  efficiency  of  a 

training  system,  in  terms  of  measurement  and  evaluation. 


ASSOCIATION  DEVICES:  Memory  aids,  techniques  which  ease  recall. 
Mnemonic  devices. 


ATTITUDE;  A persisting  state  of  a person  that  influences  his  choice 
of  action. 


ATTITUDE  MEASURE:  An  instrument  designed  to  gather  information  about 

how  people  feel  toward  a particular  object.  This  could 
include  liking  or  disliking  subject  matter,  usefulness 
of  a medium,  or  opinions  about  the  medium. 


AUDIO-ONLY  PROGRAM:  A production  which  does  not  contain  any  video 
or  pictures;  for  example,  a record  or  radio  program. 


AUDIO  PRODUCER:  Prepares  tape  recordings  and  produces  audio  programs. 
The  audio  producer  combines  narration,  music,  and 
other  sound  effects  in  the  production  of  an  audio  pro- 
gram. 


AUDIOVISUAL  MEDIA:  Refers  to  any  device  such  as  television  or  film 

which  is  both  seen  and  heard. 


BASELINE  DATA:  Valid  and  reliable  information  about  the  current 

level  of  performance  of  the  intended  student  population. 
This  data  can  be  used  to  confirm  the  need  to  develop  new 
instruction,  or  can  be  used  as  a comparison  in  ascer- 
taining differences  between  students'  performance  be- 
fore and  after  instruction. 


BEHAVIORAL  ATTRIBUTES:  Qualities  or  activities  that  characterize  an 

object  or  process.  Behavioral  attributes  characterize 
each  category  of  learning. 


BLOCK  SCHEDULING:  Mode  of  instruction  whereby  all  students  receive 

the  same  instruction  at  the  same  time. 
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.^LOCKING: 
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Refers  to  the  process  of  defining  and  illustrating  the 
different  camera  movements  and  can«ra  shots  in  a tele- 
vision or  film  script.  A blocked  script  may  also 
contain  directions  as  to  the  movanent  of  actors  as  well 
as  scenery  changes. 


CHECKLIST:  Job  performance  aid  which  lists  the  elements  of  a task 

in  the  sequence  of  execution.  The  job  holder  places 
a check  beside  each  ele.r)ent  as  it  is  accomplished,  thus 
insuring  that  the  task  is  completed. 


CHRONOLOGICAL  ORDER:  Arranging  content  in  order  from  one  topic  to 
another  based  on  when  they  occurred  In  time. 


COMMON-FACTOR  LEARNING  OBJECTIVES:  Refers  to  learning  objectives  that 
are  identical,  or  that  have  identical  action  words  and 
similar  objects  of  the  action  in  the  learning  objective 
statement. 


COMPARATIVE  SEQUENCE:  Sequencing  which  starts  with  familiar  topics 
and  goes  to  unfamiliar  ones. 


COMPLEXITY  CRITERION:  In  media  selection,  the  degree  of  complexity 
required  of  instructional  materials  in  order  to  ade- 
quately train  students  to  meet  learning  objectives. 


COMPUTER  MODELS  TECHNIQUE;  Occurs  during  the  simulation  of  an  opera- 
tional system;  involves  having  a computer  simulate  the 
major  operations  of  the  system,  under  a variety  of  condi- 
tions. 


CONDITIONS:  Occurs  in  terminal  learning  objectives;  describes  what 
is  presented  to  the  student  in  order  to  accomplish  the 
specified  action,  that  Is,  it  describes  the  important 
aspects  of  the  performance  environment. 


CONTIGUITY:  Refers,  in  learning,  to  the  principle  that  events  which 

occur  closely  together  become  associated  by  the  learner. 


CONTINGENCY  MANAGEMENT;  The  establishment  of  a set  of  procedures  by 

which  trainees  are  required  to  perform  a certain  amount  of 
work  or  to  achieve  certain  objectives  before  engaging  in 
activities  that  are  preferred  by  the  trainee  (e.g.,  recrea- 
tion, a break,  or  a more  desirable  training  event). 
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COURSE  DOCUMENTATION:  Information  describing  the  current  content 

of  a course  (instructional  materials,  tests,  instructor's 
manual,  evaluation  plan,  student's  manual)  and  its  develop- 
mental history  (job  analysis,  criteria  for  selecting  tasks 
for  training,  previous  revisions). 


CLUSTERING:  A process  of  organizing  many  tasks  into  groups  for  the 
purpose  of  deciding  upon  the  optimal  instructional  set- 
ting mix  for  that  group  of  tasks. 


CRITERION-REFERENCED  TEST:  Measures  what  an  individual  can  do  or 

knows,  compared  to  what  he  must  be  able  to  do  or  must 
know  in  order  to  successfully  perform  a task.  Here  an 
individual's  performance  is  compared  to  external  cri- 
teria or  performance  standards  which  are  derived  from  an 
analysis  of  what  is  required  to  do  a particular  task. 


CRITICAL  CUE:  Cue  which  must  be  correctly  interpreted  by  the  student  be- 

fore we  can  correctly  perform  the  associated  task. 


CRITICAL  SEQUENCE:  Sequencing  of  topics  or  objectives  according  to 
their  importance. 


CUE:  A word  or  other  signal  that  initiates  or  guides  behavior;  a 

prompt . 


CUT-OFF  SCORE:  Minimum  passing  score. 


DATA:  Collection  of  facts  or  numerical  values  resulting  from  observa- 

tions of  situations,  objects,  or  people. 


DATA  COLLECTION  PLAN:  An  outline  of  the  procedures  and  techniques 
that  will  be  used  to  gather  information  for  any  specific 
purpose. 


DATA  RECORDING  PLAN:  Method  of  tabulating  background  responses  and 
test  data. 


DECAY  RATE:  The  amount  of  time  it  takes  a trainee  to  forget  what  he 
has  learned  in  school.  If  the  decay  rate  is  high  then  a 
trainee  should  not  receive  instruction  in  a specific 
task  until  shortly  before  he  will  actually  perform  it. 
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DECISION  TREE.  Flowchart;  graphic  representation  of  the  sequence  of  a 
specific  activity  or  operation. 


DELIVERY  SYSTEM:  Any  method  containing  plans  and  procedures  for  the 
presentation  of  instruction.  Platform  instruction,  tele 
vision,  FOJT,  and  STEPs  are  all  delivery  systems. 


DEPENDENT  RELATIONSHIP:  Occurs  when  skills  and  knowledges  in  one 
learning  objective  are  closely  related  to  those  in  the 
other  learning  objective.  In  order  to  master  one  of  the 
learning  objectives,  ft  is  first  necessary  to  learn  the 
other. 


DOWNTIME;  Refers  to  the  period  of  time  when  equipment  is  inoperable. 


DUTY:  One  of  the  major  subdivisions  of  work  performed  by  one  indi- 

vidual. One  or  more  duties  constitute  a job. 


DUTY  TITLE:  Categorizes  groups  of  tasks  under  identifiable  head- 
ings to  help  in  the  organizing  of  lists  of  tasks. 


EMPIRICALLY  BASED  REVISION:  Revision  based  on  the  results  of  test 

data  and  the  collection  of  other  types  of  quantitative 
information. 


ENTRY  BEHAVIOR:  The  skill,  knowledge,  and/or  attitude  required  be- 

fore beginning  a new  segment  of  instruction;  also  may 
refer  to  the  capability  a person  has  prior  to  new 
learning. 


ENTRY  SKILLS:  Specific,  measurable  behaviors  that  have  been  determined 

through  the  process  of  analysis  of  learning  require- 
ments to  be  basic  to  subsequent  knowledge  or  skill  in  the 
course. 


ENTRY  SKILLS  TEST:  A measurement  instrument  designed  to  determine  if 

a student  already  possesses  certain  skills  or  knowledge 
needed  as  a prerequisite  before  undertaking  new  instruc- 
tion. 


ENTRY  TEST:  Contains  items  based  on  the  objectives  that  the  intended 

students  must  have  mastered  in  order  to  begin  the  course. 
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ERR(MR  OF  HALO:  Occurs  when  an  observer  sometimes  allows  his  rating 
of  performance  to  be  influenced  by  his  general  impres- 
sion of  a person. 


ERRORS  OF  LOGIC:  Occur  when  tnw  or  nwre  traits  are  being  rated.  It 
is  present  if  an  observer  tends  to  give  similar  ratings 
to  traits  which  & tt  necessarily  go  together.  The 
traits  are  relateu  only  in  the  mind  of  the  person  making 
the  error. 


ERRORS  OF  STANDARD:  Occur  when  observers  tend  to  rate  performers  too 
high  or  too  low  because  of  differences  in  their  standards. 


EVALUATICWi:  The  process  of  interpreting  the  results  of  measurement 
data  (e.g.,  tests,  JPMs)  for  the  purpose  of  making  a 
judgment  or  decision  on  the  instruction  or  on  the  success 
of  a trainee. 


EVALUATION  CRITERIA:  The  measures  used  to  determine  the  adequacy 
of  performance. 


EVALUATION  PLAN:  A method  or  outline  of  what  set  of  procedures  will 
be  used  to  gather  data  and  information  for  the  purpose 
of  assessing  a course  of  instruction. 


EXTERNAL  CUES:  signals  for  action  that  exist  outside  of  the  student 

(conditions,  features,  or  characteristics  of  the  job 
environment  that  trigger  action). 


FALSE  NEGATIVE;  Occurs  when  a person  can  perform  the  task  but 
receives  a failing  score  on  the  test. 


FALSE  POSITIVE:  Occurs  when  a person  cannot  perform  the  task  but 
receives  a passing  score  on  the  test. 


FEEDBACK:  The  return  of  information.  Information  on  student  per- 
formance is  "fed"  back  to  the  student  so  that  he  can 
improve  that  performance;  to  the  instructional  designer 
so  that  he  can  improve  materials  and  procedures  on  the 
basis  of  student  needs;  to  the  management  systea  so  it 
can  monitor  the  internal  and  external  integrity  of  the 
instruction  and  make  appropriate  revisions.  Or,  refers 
to  the  flow  of  data  or  information  from  one  step  in 
the  ISD  Model  to  others. 
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FOJT--FORMAL  ON-THE-JOB  TRAINING:  This  type  of  training  takes  place 

in  the  actual  work  situation. 


FOLLOW-UP  ACTIVITIES:  The  work  events  that  occur  after  a course  of 

instruction  has  been  completed. 


FOPJWIVE  EVALUATION:  The  iterative  process  of  developing  and 

improving  instructional  materials  and  procedures. 


FIDELITY;  Refers  to  how  well  the  actions,  conditions,  cues,  and 
standards  of  the  JPM  approximate  those  of  the  task. 


FIELD  USER  NEEDS:  The  general  and  specific  duties  that  will  have 

to  be  taught  to  the  trainee  if  he  is  to  be  able  to 
adequately  perform  iri  a real  world  environment. 


FIPST  DRAFT  MATERIALS:  Any  materials  (book,  film,  etc.)  which  are 
not  yet  committed  to  their  final  form.  First  draft 
refers  to  the  fact  that  the  materials  are  still  in 
'rough'  form  and  will  be  revised  on  the  basis  cf  test 
results  and  other  data. 


FLOWCHART:  A graphic  representation  of  the  sequence  of  a specific 

activity  or  operation;  decision  tree. 


FRONT  END  ANALYSIS:  Refers  to  job  analysis,  selection  of  tasks  for 

training,  and  development  of  JPMs. 


FIXED  SEQUENCE:  Refers  to  elements  that  are  always  done  in  the  same 

order. 


GRAPHIC  ARTIST:  Designs  and  prepares  a wide  variety  of  visual  illu- 

strations such  as  graphs,  charts,  and  diagrams. 


GRAPHIC  SCALE:  Measurement  de\ice  which  includes  some  type  of  number 

line  on  which  students  indicate  their  attitude  toward 
a social  object. 


GO  NO-GO:  Pass-fail;  criterion  of  evaluation  whereby  student  can- 

not be  "partially  correct".  He  is  either  100%  correct 
(go)  or  incorrect  (no-go). 
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GROUP  MANAGEMENT  PLAN:  Arrangenent  whereby  instruction  is  scheduled 
and  conducted  for  groups  instead  of  individuals. 


GROUP  TRAINING:  A group  of  people  gathered  together  for  the  purpose 
of  receiving  information  or  instruction  in  the  performance 
of  some  specific  task. 


HARD  DATA:  A direct  and  precise  measure  of  a specific  performance. 

A JPM  is  an  example  of  hard  data  while  an  attitude  question- 
naire is  a less  direct  nwasure,  providing  soft  data. 

HIGH  DENSITY  SIGNAL:  A signal  containing  many  cues.  A low  density 
signal  contains  few  cues. 


INDEPENDENT  RELATIONSHIP:  Occurs  when  skills  and  knowledges  in  one 

objective  are  unrelated  to  those  in  the  other  objective. 
Mastering  one  of  the  objectives  does  not  simplify  the 
other. 


INDICATOR  BEHAVIOR;  Refers  to  that  behavior  that  indicates  the 
presence  of  a specific  attitude. 


INDIVIDUALIZED  INSTRUCTION:  Refers,  in  the  ISD  Model,  to  a management 
scheme  which  permits  individual  characteristics  of 
trainees  to  be  a major  determinant  of  the  kind  and  amount 
of  instruction  given.  Here,  it  nearly  always  implies 
some  form  of  self-pacing. 


INSTALLATION  SUPFjRT  SCHOOLS:  Organized  and  operated  by  individual 
units  or  commands  to  meet  local  training  requirements. 


INSTRUCTIONAL  CONDITIONS:  The  amount  of  participation  which  the 
instruction  requires  of  the  learner.  Instructional 
conditions  may  be  active  (the  learner  produces  or  prac- 
tices) or  passive  (the  learner  sits  and  listens). 


INSTRUCTIONAL  DESIGNER:  Person  who  designs  and  develops  a program  or 

course  of  studies  based  on  a systenatic  analysis. 


INFORMATION:  Knowledge;  the  facts,  names,  labels,  and  larger  bodies 

of  knowledge  that  are  necessary  for  successful  job 
performance . 
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INSTRUCTIONAL  MANAGEMENT  PLAN:  The  specification?  for  the  schedul iny. 

instruction  and  evaluatiofi  of  trainee.,  tovjurd  the  yoj  I of 
course  completion. 


INST.RUCTIONAL  PROGRAM:  The  development  of  various  materials  (books, 

audiovisual  productions,  etc.)  designed  to  achieve  a 
specific  training  goal. 


INSTRUCTIONAL  SETTING:  The  vehicle  through  which  a trainee  who 

initially  is  ;iot  able  to  perform  a task  becomes  proti- 
cient  in  performing  the  task;  for  example,  perfonuance 
aids,  self-teaching  exportable  packages,  formal  on-job 
training,  installation  support  scfiools,  and  resident 
schools. 


INSTRUCTIONAL  SUPPORT;  Learning  resources;  different  kinds  of 

material,  number  of  instructors,  amount  of  time,  etc. 
which  will  contribute  to  the  learning  situation. 


INSTRUCTIONAL  SYSTEM:  The  total  effort,  distinct  from  the  operating 

system  by  location,  authority,  or  mission,  that  is  con- 
cerned with  the  preparation  of  individuals  to  serve  the 
operating  system. 


INTERNAL  CUES:  Internal  biological  signals  that  initiate  or  guide 

behavior. 


INTERNAL  EVALUATION:  Assessment  of  the  effectiveness  of  an  instruc- 

tional program  in  terms  of  student  performance  on 
stated  terminal  learning  objectives. 


JOB:  The  duties  and  tasks  performed  by  a single  worker  constitute 

his  job.  If  identical  duties  and  tasks  are  performed  by 
several  individuals,  they  all  hold  the  same  job.  The 
job  is  the  basic  unit  used  in  carrying  out  the  personnel 
actions  of  selection,  training,  classification,  and 
assignment. 


JOB  ANALYSIS:  The  basic  method  used  to  obtain  a detailed  listings  of 

duites,  tasks,  and  elements  necessary  to  perform  a 
clearly  defined,  specific  job,  involving  observations  of 
workers  and  conversations  with  those  who  know  the  job, 
in  order  to  describe  in  detail  the  work  involved,  includ- 
ing conditions  and  standards. 
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JOB  FIDELITY;  The  degree  to  which  a testing  situation  truthfully 
and  accurately  reflects  the  job  situation. 


JOB  PERFORMANCE  ME.ASURES:  Tests  that  are  used  to  evaluate  proficiency 

of  a job  holder  on  each  task  he  performs. 


JOB  PERFORMANCE  TEST:  Test  used  to  determine  whether  or  how  well  an 
individual  can  perform  a job.  In  may  include  either  all 
of  the  job  performance  measures  for  a particular  job  or 
a subset  of  the  job  perfoimance  measures. 


JPA--J0B  PERFORMANCE  AID:  A checklist,  instruction  sheet,  or  other 
device  that  offers  a possible  alternative  to  training 
rather  than  an  actual  method  of  training;  they  are  de- 
veloped to  eliminate  or  minimize  training  requirements 
for  some  tasks. 


KNOWLEDGE  OF  RESULTS:  Feedback;  information  provided  to  the  student 

indicating  the  correctness  of  his  response.  Evaluative 
knowledge  of  results  indicates  what  a student  is  doing 
right  and  what  he  is  doing  wrong.  Comparative  knowledge 
of  results  indicates  how  the  student's  response  compares 
to  the  objective  or  standard  established  by  the  instruc- 
tor. 


LEARNER  CHARACTERICTICS:  T.he  traits  possessed  by  learners  that  could 

affect  their  ability  to  learn  (e.g.,  age,  I.Q.,  reading 
level , etc.). 


LEARNING  ACTIVITY:  The  specific  behaviors  a student  performs  during  j 

a particular  episode  of  learning.  i 

J 
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LEARNING  ANALYSiS:  A procedure  to  identify  subelements  that  must  be 
learned  before  a person  can  achieve  mastery  of  the 
performance. 

LEARNING  CATEGORY:  A division  of  learning  behavior.  All  learning 
may  be  classified  into  one  of  four  learning  categories: 
mental  skill,  physical  skill,  information,  or  attitude. 


LEARNING  EVENT:  The  immediate  outcome  of  a learning  activity. 
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LEARNING  GUIDELINES:  Statements  which  specify  the  learning  events 

and  activities  appropriate  to  specific  instruction. 
Learning  guidelines  combine  to  form  learning  sub- 
categories. 


LEARNING  HIERARCHY:  Graphically  portrays  the  relationships  among 
learning  tasks  in  which  some  tasks  must  be  mastered 
before  others  can  be  learned. 


LEARNING  OBJECTIVE;  Describes  precisely  what  is  to  be  learned  in 
terms  of  the  expected  student  performance  under  speci- 
fied conditions  to  accepted  standards.  These  learning 
objectives  identify  the  mental  skills,  information, 
attitudes,  or  physical  skills  that  are  required  to  per- 
form the  terminal  learning  objective. 


LEARNING  RESOURCE  CENTER;  Library  containing  instructional  materials 
and  areas  for  viewing  and  study. 


LEARNING  STEP:  Occurs  when  learning  objectives  are  broken  down  into 
smaller  parts. 


LEARNING  SUB-CATEGORY:  A division  of  a learning  category. 


LEARNING  TASK  ANALYSIS;  Procedure  used  in  the  domain  of  intellectual 
skills  to  Identify  prerequisite  tasks  that  must  be 
learned  before  a person  can  learn  a given  task. 


LINK  TRAINER:  Mechanical  training  device  which  simulates  the  cock- 

pit of  an  aircraft. 


RESPONSE  PIAS:  Tendency  to  favor  a certain  response  over  others. 


MANAGEMENT  PLAN:  Program  for  the  assignment,  monitoring,  and  assess- 

ment of  the  personnel,  materials,  and  resources  dedi- 
cated to  a specific  mission,  operation,  or  function. 


MASTERY:  In  terms  of  learning,  refers  to  meeting  all  of  the  specified 

minimum  requirements  for  a specific  performance.  Criteria 
for  mastery  are  defined  in  the  design  phase  of  the  ISD 
Model . 
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MEAN;  Arithmetic  average  calculated  by  adding  up  all  scores  and 
dividing  by  the  number  of  scores. 


MEASUREMENT:  Consists  of  rules  for  assigning  numbers  to  objects  to 

represent  quantities  of  attributes. 


MEASUREMENT  ERRORS:  Incorrect  procedures  carried  out  during  the 

measurement  process  which  invalidate  the  results.  These 
errors  resuH  from  unfounded  assutntions  nade  by  judges 
or  raters. 


MEASUREMENT  PROCESS:  The  operations  involved  in  determining  the 

amount  of  an  attribute  (e.g.,  skill,  knowledge,  or  atti- 
tude) possessed  by  a student. 


MEDIA:  Means  for  presenting  instructional  material  to  learners;  for 

example,  books,  audiotapes,  and  filmstrips. 


i 
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MEDIA  ALTERNATIVE:  A form  of  instructional  material  that  contains 

the  stimulus  criteria  required  by  a specific  learning 
activity. 


MtOIA  MIX;  Combination  of  different  media  used  to  present  a unit  of 
instruction. 


MEDIA  POOL:  All  of  the  media  options  suitable  for  a given  unit  of 

iristruction.  The  final  media  choice  is  drawn  from  the 
media  pool. 

MEDIA  SELECTION:  Is  the  major  means  of  determining  how  instruction  is 

to  be  packaged  and  presented  to  the  student. 


MENTAL  SET:  A preparatory  mental  adjustment,  or  readiness,  for  a 

particular  type  of  experience. 


MENTAL  SKILLS:  Those  processes  of  identifying,  classifying,  using 

rules,  and  solving  problems  that  involve  active  mental 
processing.  Mental  skills  imply  the  capability  of 
applying  the  learning  to  some  situation  and  demonstrating 
the  mental  skill,  such  as  thinking,  creating,  and 
analyzing. 
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MNEMONICS:  Methods  which  make  information  easier  to  remember;  memory 
aids. 


MODE  OF  INSTRUCTION:  Method  of  scheduling  materials  presentation.  The 
instructional  mode  may  be  individualized  (self-pacing)  or 
group  (block  scheduling). 


MODULE:  An  individualized  self- instructional  package  usually  con- 
taining all  the  necessary  materials  a learner  needs  to 
meet  some  or  part  of  a terminal  learning  objective. 


MULTIMEDIA  PACKAGE:  Self-contained  instruction. mit  in  more  than 

one  medium. 


NARRATION:  Is  the  voice  overheard  on  an  audiovisual  program. 


NARRATOR:  Is  the  person  whose  voice  is  heard  describing  or  commenting 

upon  the  content  of  a film,  television  program,  etc. 


NUMERICAL  SCALE:  Measurement  device  which  associates  verbal  descrip- 
tions of  social  objects  with  numbers  and  requires  students 
to  indicate  their  attitudes  by  marking  the  appropriate 
number . 


OBSERVATION  INTERVIEW:  Job  holder  is  observed  in  the  job  environment 

performing  all  or  a substantial  part  of  the  job;  the  job 
holder  performs  the  job  while  the  analyst  ask  questions. 


OFF-LINE:  Refers  to  any  activity  which  does  not  take  place  as  part 

of  the  regular  production  process. 


OVERLEARNING:  Refers  to  the  continual  practice  on  a learning  task  by  a 

person  who  has  correctly  performed  the  task. 


PEER  TUTORING:  A form  of  instruction  in  which  students  at  the  same 
or  more  advanced  level  of  knowledge  provide  instruction 
to  students  at  the  same  or  lower  level  of  knowledge  on 
the  specific  objectives  under  consideration.  Peer  tutors 
are  not  members  of  the  existing  instructional  establish- 
ment. 
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PERFORMANCE  EVALUATION:  The  gathering  of  data  to  specifically 

determine  the  success  of  students  on  a specific  task, 
as  a result  of  a training  program. 


PERFORMANCE  MEASURES:  The  absolute  standard  by  which  a job  perfor- 

mance is  judged.  A performance  measure  is  the  inven- 
tory of  job  tasks  with  each  performance  objective. 


PERSE  ERATE:  Continue  an  activity  until  it  is  completed,  regardless 

of  the  difficulty,  or  the  appropriateness  of  the  solu- 
tion technique  to  the  problem. 


PERT— PROGRAM  EVALUATION  REVIEW  TECHNIQUE:  PERT  is  a method  of  moni- 

toring the  flow  of  a large  project  by  breaking  it  down 
into  small  individual  activities  and  assigning  each 
activity  a specified  amount  of  time  for  completion. 


PHYSICAL  SKILLS:  Specified  muscular  activities  for  accomplishing 

a goal . 


POST  FEEDBACK  DELAY:  The  pause  which  follows  the  presentation  of 
feedback.  This  allows  time  for  the  correct  response  to 
"sink  in." 


POSTTEST:  A test  administered  after  the  completion  of  instruction  to 

assess  whether  a student  has  mastered  the  objectives  of 
the  course  or  unit. 


PREDICTIVE  VALIDITY:  The  ability  of  a test  score  to  accurately  fore- 

cast future  performance. 


PREDIFFERENTIATION  OF  STIMULI:  Pointing  out  the  distinguishing 

features  of  an  object  and  explaining  the  differences 
between  them. 


PRETEST:  Administered  prior  to  instruction  to  determine  how  much 

the  student  already  knows. 


PROCESS  EVALUATION:  An  early  stage  in  ISD  development  that  identifies 

which  steps  in  the  model  will  be  used  for  the  course  under 
development.  The  purpose  of  the  process  evaluation  is  to 
describe  and  document  the  actual  developmental  process  for 
this  particular  instruction. 
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PROCESS  STANDARDS:  Refers  to  the  conditions  which  must  be  satisfied  for 

a job  to  be  successfully  completed.  Process  standards  refer 
to  sequence,  accuracy,  speed  of  performance,  and  complete- 
ness. 


PROGRAWiED  INSTRUCTION:  Instructional  materials  which  present  subject 

matter  in  a series  of  small  sequential  units  which  require 
responses  from  the  student. 


PRWPT:  A word  or  other  signal  that  initiates  or  guides  behavior; 
a cue. 


QUALITY  CONTROL:  Process  of  measuring  and  evaluating  in  order  to  main- 

tain course  standards  through  adjustments  in  instructional 
materials  or  procedures. 


QUALITY  CONTROL  DATA:  Information  which  reflects  the  degree  of  success 

achieved  by  a system  or  operation. 


RANDOM  SELECTION:  Choosing  people  or  objects  at  random  rather  than 
according  to  some  systematic  plan. 


RANK  ORDER:  The  assignment  of  ranks  to  students.  This  could  refer  to 

groups,  such  as  the  top  10%,  or  simply  listing  each  stu- 
dent from  highest  to  lowest.  Rank  ordering  is  appro- 
priate when  there  is  a need  to  select  the  fastest,  the 
most  accurate,  oi  the  best  producer. 


RATING  ERRORS:  Errors  of  standards,  ratio,  and  logic. 


RATING  SCALE:  A measurement  device  in  which  a student  must  choose  a 

response  from  a range  of  choices  arranged  in  a continuum 
from  low  to  high  or  good  to  bad,  etc. 


REGULATIONS:  Rules  for  appropriate  conduct  and  behavior. 


RELIABILITY:  The  consistency  with  which  a test  measures  the  amount  of 
student  achievement. 


RESIDENT  SCHOOLS:  These  schools  are  designed  to  meet  service-wide 

training  requirements. 
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REVISION  PLAN;  A detailed  outline  of  the  procedures  to  be  taken 
to  modify  the  structure  or  coi.tent  of  a course. 


REWARD  SEQUENCE:  Scheduling  the  more  pleasant  activity  to  follow 

the  less  pleasant  activity;  can  be  used  to  provide 
a reward  for  completion. 


SAMPLE:  A portion  or  small  segment  of  the  students  for  whom  instruc- 
tion is  designed. 


SAMPLING  PLAN:  Procedure  for  selecting  a small  but  representative 

group  from  a larger  population. 


SCALE;  In  media  selection,  some  materials  must  represent  actual 
objects  and  accurately  represent  the  dimensions  of 
those  objects.  A model  may,  for  example,  be  full 
scale,  half  scale,  or  on  a 1 to  10  scale  with  the 
actual  object. 


SELF  PACING:  Mode  of  instruction  whereby  each  student  works  through 
the  instructional  materials  at  his  own  rate  of  speed. 


SELF"PACED  MANAGEMENT  PLAN:  Arrangement  whereby  instruction  is 

scheduled  and  conducted  for  individual  students  rather 
than  groups  of  students. 


SELF-TEACHING  EXPORTABLE  PACKAGES:  Self  instructional  study  units; 

generally  sent  to  the  student  wherever  he  is  stationed. 


SEQUENCING:  Ordering  instruction;  proper  sequencing  allows  the 

learner  to  make  the  transition  from  one  skill  or  body 
of  knowledge  to  another,  and  assures  that  supporting 
skills  and  knowledge  are  acquired  before  dependent 
performances  are  introduced. 


SHAPING:  Gradually  changing  a student's  behavior  until  it  is  correct. 


SIGNAL:  Cue  that  initiates  and  directs  activity. 


SIMULATION: 
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Any  change  from  reality  or  any  imitation  of  reality. 
Three  type?  are  common:  simulating  part  of  the  system, 

simulating  the  operation  of  the  systam,  and  simulating 
the  environment  in  which  the  system  will  operate. 


SIMULATORS:  Machines  or  processes  designed  to  provide  training  which 

will  have  high  positive  transfer  to  the  real  world  equip- 
ment or  situation.  Siirmlators  are  ordinarily  cheaper, 
safer,  or  more  available  than  the  actual  situation  or 
equipment. 


SLIDE-TAPE:  A combination  of  visual  slides  and  an  audio  tape  syn- 

chronized sc  that  the  audio  describes  the  content  of 
the  slides. 


SOFT  DATA:  Obtained  from  attitude  or  opinion  surveys.  This  data  is 
not  as  reliable  as  hard  data. 


STA.NDAROS:  Occurs  in  terminal  learning  objectives  or  learning 

objectives;  describes  the  criterion  or  standard  of  per- 
formance winch  must  be  attained. 


STIMULUS  CRITERIA:  Those  basic  qualities  or  capabilities  of  a 

medium  that  are  required  to  carry  out  the  intent  of  the 
learning  activity;  for  example,  visual  images,  motion, 
color,  and  sound. 


STORYBOARD:  A collection  or  series  of  small  pictures  which  describe 
the  action  and  content  that  will  be  contained  in  an  audio 
visual  or  visual -only  production.  A sequence  of  these 
small  pictures  comprise  a storyboard. 


SUBJECT  MATTER  EXPERT:  A person  who  has  professional  skill  in  the 

performance  of  some  job  and  who  is  consulted  by  an  in- 
structional designer  in  the  process  of  job  task  analysis. 


SUPPORTIVE  RELATIONSHIP:  Occurs  when  skills  and  knowledges  in  one 
objective  have  sane  relationship  to  those  in  the  other 
objective;  the  learning  involved  in  mastery  of  one  learn- 
ing objective  transfers  to  the  other,  making  learning 
involved  in  the  mastery  of  the  other  easier. 
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SYMBOL:  Anything  that  stands  for  or  represents  something  else.  A 

plus  sign  (+)  is  a symbol  for  the  mathe''.atical  operation 
of  addition. 


SYSTEM  MASTER  PLAN:  Control  document  used  to  coordinate  the  develop- 

ment and  implementation  of  an  instructional  program. 


SYNCHRONIZING  PULSE:  An  audible  or  inaudible  sound  used  to  coordinate 
the  audio  and  video  portions  of  a slide-tape  program  so 
that  audio  and  video  (i.e.,  slide  and  narration)  are 
coordinated. 


SYSTEMS  APPROACH:  A generic  term  referring  to  the  orderly  process  of 

analysis,  design,  development,  evaluation,  revision,  and 
operation  of  a collection  of  interrelated  elements. 


TALK-THROUGH  TECHNIQUE:  Occurs  during  the  simulation  of  an  operational 
system;  involves  talking  through  each  operation  in  the 
new  system  to  determine  decisions  and  contingencies. 


TARGET  POPULATION:  The  pool  of  potencial  entrants  to  training  for 

which  instructional  materials  are  designed  and  tried  out. 


TASK  DELAY  TOLERANCE:  A measure  of  how  much  delay  can  be  tolerated 
between  the  time  the  need  for  task  performance  becomes 
evident  and  the  time  actual  performance  nwst  begin. 


TASK:  Formed  in  clusters  which  make  up  duties.  A task  is  the  lowest 

level  of  behavior  in  a job  that  describes  the  performance 
of  a meaningful  function  in  the  job  under  consideration. 


TASK  INVENTORY:  List  th^t  itemizes  aP  of  the  tasks  that  make  up 

a selected  duty. 


TASK  LEARNING  DIFFICULTY:  Refers  to  time,  effort,  and  assistance  re- 

quired by  a student  to  achieve  performance  proficiency. 


TASK  STANDARD:  A statement  of  how  well  a task  must  be  performed. 
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TASK  STATEMENT:  A statement  of  highly  specific  action  which  has  a 

verb  and  object;  for  example,  sort  mail. 


TECHNICAL  ORDERS:  Military  regulations  which  deal  with  the  specific 
nature  of  technical  materials  and  equipment. 


TERMINAL  LEARNING  OBJECTIVE;  Derived  from  job  performance  measures, 
TLOs  are  to  be  attained  during  training.  TLOs  are  broken 
down  into  their  component  parts  which  are  documented  as 
learning  objectives  which  may  be  further  divided  into 
learning  steps.  Each  TLO  contains  actions,  conditions, 
and  standards. 


TESTS:  Any  device  or  technique  used  to  measure  the  performance  of  a 

student  on  a specific  task  or  subject  matter. 


TESTING  CONSTRAINTS:  Limitations  such  as  time,  money,  personnel, 

facilities,  and  ether  resources,  which  prohibit  job 
performance  meacures  from  being  identical  to  the  tasks 
they  measure. 


TRADE-OFFS:  In  any  systematic  approach  to  instruction,  it  is 

necessary  to  make  compromises  between  what  is  desirable 
and  what  is  possible.  Ordinarily,  these  decisions  in- 
volve increases  or  decreases  in  time,  money,  facilities, 
equipment,  or  personnel.  Training  aids  and  simulators 
represent  examples  of  trade-offs. 


TRAINER  APPRAISAL  KIT:  A package  of  instructional  materials 

designed  to  provide  a course  instructor  with 
practice  in  the  preparation,  presentation,  and  valida- 
tion of  instruction. 


TRAINING;  The  teaching  of  job  skills.  It  can  take  a number  of  forms 
such  as  self -teaching  exportable  packages,  training 
manuals,  individual  learning  packages,  FOJT,  or  group 
training. 


TRAINING  SETTING  CRITERIA:  In  media  selection,  the  options  that 

training  must  be  either  small  group,  lar;jt;  group,  indi- 
vidualized at  a fixed  location,  or  individualized  inde- 
pendent of  location. 
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TRYOUT:  Practice  test;  the  purpose  is  to  make  the  tryout  as  realistic 
as  possible  by  eliminating  as  many  sources  of  unreliability 
as  possible. 


UNDERTRAIN:  Provide  inadequate  training  that  does  not  prepare  a 
student  to  meet  regular  job  performance  requirements. 


VALIDATION:  A process  through  which  a course  is  revised  until  it 

is  effective  in  realizing  its  instructional  goal. 


VALIDATION  DOCUMENTATION:  A report  which  describes  in  detail  how 
a specific  course  of  instruction  was  validated  and  for 
what  target  population. 


VALIDATION  PROCESS:  Testing  instructional  materials  on  a sample  of 
the  target  population  to  insure  that  the  materials  are 
effective. 


VALIDITY:  The  degree  to  which  a test  measures  what  it  claims  to 
measure. 


VALUE  ENGINEERING;  Refers  to  the  proces.s  of  designing  equipment 

or  instruction  to  meet  but  not  exceed  the  required  out- 
comes. Ordinarily,  it  refers  to  the  elimination  of 
features  or  instructional  objectives  that  have  not  been 
demonstrated  to  be  positively  necessary. 


VIGILANCE  LEVEL:  General  degree  of  watchfulness  or  attentiveness 

to  what  may  come. 


VISUAL  FORM:  In  media  selection,  refers  to  whether  alphanumeric  or 

pictorial  characteristics  are  required  in  a learning 
situation. 


VISUAL  SPECTRUM:  The  type  of  color  required  of  instructional 

materials.  Some  must  be  with  full  color,  others  may 
be  with  black  and  white  or  shades  of  grey. 


WITHIN-COURSE  TESTS:  Administered  during  a course  of  instruction 

to  assure  that  all  students  are  "keeping  up"  with  the 
learning  objectives. 
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WORK  ELEMENTS:  The  element  is  the  smallest  component  in  the 

structure  of  a job.  Elements  combine  to  form  a task, 
tasks  combine  to  form  a duty,  and  duties  combine  to 
form  a job. 
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category 11,  III:  66-/0,  103-104 
affective  component,  III:  66-67, 
103 

behavioral  component.  III:  66, 
70,  104 

cognitive  component.  III:  66, 
67-70,  104 

Audio  producer  (production  person- 
nel), III:  228 

Audio-visual  production.  III:  235- 
244 


Baseline  data  collection,  V:  25-26, 
70-71 

Behavioral  component:  attitude 
learning.  III:  66,  70,  104 
Block  scheduling  (group  instruc- 
tion plan).  III:  124-125 


Classifying:  learning  sub- 
category 2,  III:  38-40,  74-75 
Clustering,  task,  I:  244-245 
Cognitive  component:  attitude 
learning.  III:  66,  67-70,  104 
Correnand  Job  Analysis,  sc'nedule  for, 
I:  58 

Common- factor  learnina  objectives, 
II:  90-92 

Comprehensive  Occupational  Data 
Analysis  Program  (CODAP), 

I:  121 


Computer-assisted  instruction  (CAI), 
III:  191 

Computer-managed  instruction,  III: 
191-19^ 

Conditions,  JPM,  I:  177-178 

on-the-job,  task,  I:  24-26,  89-93 
Contiguity,  III:  41 
Contingency-managed  instruction, 

III:  187-189,  215 

Contract,  student  (or  perfo>^mance) , 
III:  189 

Cues,  JPM,  I:  179 

prompts  for  student  learning.  III: 
6-8 

task,  I:  25,  27-28 


Data  collection, 

for  external  evaluation,  V:  68-77 
for  internal  evaluation,  V:  1-8, 
29-46,  53-62 
for  JPMs,  1:45-46 
for  rating  tasks,  I:  133-139 
Data  collection  plan  for  job  analysis, 
alternate  or  short-cut,  I:  47, 

77-78 

form  preparation,  I:  49-51,  59 
group  interviews,  I:  40 
individual  and  observation  inter- 
views, I:  36-37,  41-46,  52-62 
jury-of-experts,  I:  39-40 
questionnaire  survey,  I:  37-39, 

45,  46,  68-73 

Decision-making:  learning  sub-category 
5,  III:  46-48,  83-86 
Decisions,  management,  Executive 
Summary:  20,  24-25,  30-32,  36-37, 
42-43,  51 , 54-55,  59,  63,  69-70, 
75-76,  81-82,  85-86,  91-92,  96-97, 
100-101,  108-109,  114-115,  119-120 
Decision  trees,  see  Algorithms 
Defense  Occupational  Specialties  (DOS), 
I:  3,  8,  214 

Delay  tolerance,  s^e  Task  delay  ... 


Preceding  page  Uaek 
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Delivery  system,  III:  213-214, 
see  a I so  Media  and  Instructional 
management  plan  and  delivery 
system 

Detecting:  learning  sub-category  4, 
III:  43-45,  78-82 
Duty,  I:  11-12 


Electronic  technician  (production 
oersonnel),  III:  229 
Element(s),  I:  17-22 

alternate-path,  I:  22-24 
fixed-sequence,  I:  22-24 
Entry  behavior,  student,  II:  62-77 
definition  of,  II:  62-63 
entry  tests  for,  II:  70-72 
outputs,  II:  76 

procedures  for  determining,  II: 
64-75 

purpose  of  establishing,  II: 
62-63 

pretesting,  II:  62,  73-75 
remedial  or  preparatory 
instruction,  II:  69 
revise/ test  procedure,  I!:  69 
tests  to  verify  assumptions 
about,  II:  63-69 

Entry  test,  development  of,  II:  72 
need  for,  II:  70-72 
Equipment-oriented  jobs,  I:  81-83 
Errors  in  measuring  and  testing, 

I:  201-205;  IV:'33 
of  halo,  I:  203-204;  IV:  33 
of  logic,  I:  204-205 
of  standard,  I:  202;  IV:  33 
Evaluation,  external,  External 
evaluation 

Evaluation,  internal,  see  Internal 
evaluati on 

Existing  courses,  analysis  of,  I: 
213-229 

development  documentation  for, 

I:  220 

evaluating  existing  job  analy- 
ses for,  I:  215-216,  221-222 
evaluating  existing  Job  Perform- 
ance Measures  for,  I:  216, 
225-226 

evaluating  tasks,  I;  223-224 
evaluating  validation  documen- 
tation for,  I:  226 
Front  End  Analysis,  I:  216-218 


Existing  courses  (continued) 
outputs,  I:  218,  227-228 
procedures  for,  I:  218-227 
rationale  for,  I:  213,  215 
revisions  indicated  by,  I:  227 
Existing  materials,  review  of,  III: 
198-220 

collection  process  for.  III: 

202-203 

modifications  determined  by.  III: 
206-208,  212-213,  216 
outputs.  III:  216-217 
procedures  for,  III:  201-216 
rationale  for,  III:  198-201 
with  reference  to  delivery  system, 
III:  213-214 

with  reference  to  learner  charac- 
teristics, III:  203-205 
with  reference  to  learning  guide- 
lines, III:  208-213 
with  reference  to  management  plan, 
III:  214-215 

External  evaluation,  V:  63-86 

baseline  data  collection,  V:  70-71 
data  collection  methods,  V:  68-69 
data  collection  procedures,  V:  69- 
77 

data  consolidation,  V:  78-81 
data  requirements,  V:  66-68 
data  sources,  V:  65-66 
documentation,  V:  84-85 
job  performance  evaluation  data 
collection,  V:  71-73 
outputs,  V:  83-85 
personal  interview  data,  V:  74-75 
planning  phase,  V:  65-69 
procedures,  V:  65-83 
purposes  of,  V:  63-65 
questionnaire  use,  V:  73-74 
recommendations,  V:  81-83 
records  of  students'  performance 
during  instruction,  V:  76-77 
report  (EXER),  V:  78,  83 
time  considerations,  V:  68 
External  Evaluation  Report  (EXER),  V; 
78,  83,  90-91,  93,  95-96,  105,  109- 
110,  112 


Feedback,  III:  6,  9-10 
Formal  On- the- Job  Training  (FOJT),  I: 
231,  237-238,  256-258;  III:  139- 
141,  260-262 


IM 


Formative  evaluation^.  Ill;  280-281 
Front  End  Analysis,  I:  216-218 


Graphic  artist  (production  person- 
nel). Ill:  227-228 


Hard  data.  V:  4 


Identifying  symbols,  learning  sub- 
category 3;  III:  41-42,  76-77 
Installation  Support  Schools  (ISS), 
I:  238,  258-259;  HI:  140-142 
Instruction,  conducting,  IV:  29-39 
course  documentation,  pre- 
instruction review  of,  IV: 
32-34 

changes  in,  IV:  36-38 
doc!:mentation  of  changes, 
problems  for,  IV;  35-37 
follow-up  activities,  IV:  37-38 
function  of  instructor,  IV:  29- 
31,  32-38 
outputs,  IV-  38 
procedures,  IV:  32-38 
test  administration,  IV;  32-33, 
34-35 

training  of  instructors,  IV:  34- 
35 

Instruction,  development  of,  III: 
221-287 

adjunct  programs.  III:  251 
aims.  Ill:  221-223 
audio-only  script  guidelines, 
III;  230-232,  233 
audio-visual  production,  III: 
235-244 

Formal  On-the-Job  Trailing 
(FOOT),  III;  260-262 
Instructor's  Guide,  III:  265- 
266 

Oob  Performance  Aids  (JPAs), 

III:  252-260 

needs  and  constraints,  III: 
224-225 

outputs,  III : 2/0 
platform  lectures.  III:  248 
pre-testing  of  draft  materials, 
III:  263-265 


Instruction,  development  of  (con- 
tinued) 

proceSurcs.  Ill:  224-269 
production  personnel , III:  227-229 
progrMMtd  Instruction,  III:  246- 
247 


resources.  III:  225-230 
Self-Teaching  Exportable  Packages 
(steps).  Ill;  248-249 
slide- tape  production.  III:  239- 
244 

Students'  Guide,  III:  269 
supplementary  instruction.  III: 
249-250 

television  program  production, 
III;  244-246 

terms  defined.  III:  286-287 
types  of  instruction.  III;  230- 
263 

user  instructions.  III:  265 
video-only  materials.  III:  232, 
234-235 

Instructional  management  plan  and 
delivery  system.  III:  105-197 
block-scheduling  for  group 
instructions  III;  124-125 
consumables  and  courseware.  III: 
136-137 

course  management.  III:  126-127 
definition.  III:  105 
equipment  and  facilities,  III: 
135-136 

for  instructors.  III:  131-135 
for  student  processing.  III;  127- 
130 


for  support  personnel.  III:  135 
function  of  guidelines.  III:  106- 
107 


group  assignment,  methods  of, 
III:  125 


management  guidelines.  III:  124- 
137 

media  costs,  III:  112 
media  mixes.  III:  107-118 
media  selection,  III;  118-124 
outputs.  III:  142-143 
procedures  for  specifying.  III: 
107-142 

program  completion,  III;  131 
self-pacing  mode.  III:  124,  125- 
126 


printed  materials,  III;  246 


System  Master  Plan  (SMP),  III: 
105.  107,  138-142 
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Instructional  management  plan,  imple 
mentation  of,  IV;  1-18 
equipment,  IV:  15 
instructor's  manual,  reviewed 
and  supplemented,  IV:  4-5 
materials,  IV:  15 
procedures,  IV:  4-lb 
space  allocation,  IV:  14-15 
staff  training,  IV:  7-12 
student  salection,  IV;  13 
student  manuals,  reviewed  and 
supplemented,  IV:  6 
time  allocation,  IV:  14 
time  relationships  within  ISD 
process,  IV:  3-4 
Trainer  Appraisal  Kit  (TAK), 

IV:  7-12 

Instructional  materials,  validation 
of.  Ill:  280-346 
attitude  measures,  III:  288, 
294-295 

data  analysis  plan.  III:  295-297, 

316 

data  collection  and  recording 
system.  Ill:  287-293,  314-316 
data  consolidating  and  analyzing, 
III:  304-308,  331 
group  trial,  administering.  III: 
328 

group  trial  procedures,  Hi:  316- 

317 

individual  trials,  plan  for  con- 
ducting instruction  for.  III: 
297-298 

individual  trials,  steps  in  con- 
ducting, Illr  298-303 
learning  objective  rating,  III: 
317-327 

nature  and  purpose  of,  III:  230- 
285 

outputs.  III:  335-339 
participant  selection  for  group 
trials.  III:  317,  328 
plan  for  group  trials,  III:  312- 
328 

plan  for  individual  trials.  III: 
285-298 

platform  instruction,  III:  329- 
331 

procedures.  III:  285-336 
revision  of  attitude  measures, 
III:  316 


Instructional  materials,  validation 
of,  (continued) 

revision  requirements,  III:  308- 
311,  332-336 

standards  and  constraints  for 
group  trials.  III;  313-314 
Instructional  methods  and  devices, 

III:  163-172 

audio-only  systems.  III:  163 
audio-visual  systems,  III;  163-156 
computer  simulation,  HI;  166-168 
print  materials.  III:  169-171 
special  and  non-standard  items, 

III:  171-172 

visual -only  systems,  III:  168-169 
Instructional  setting(s),  selection 
of,  I;  230-264 

assignment  of  tasks,  I;  233-263 
clustering  tasks,  I:  244-245 
cost  analysis  for,  !:  ?43’?42 
definitioii  of  instructional 
setting,  I:  230-231 
documentation  for,  I:  262-263 
Formal  On- the- Job  Training  (FOOT), 

I:  231,  237-238,  256-258 
group  training,  I:  239 
inputs,  I:  231-232 
Installation  Support  Schools  (ISS), 
I;  231 , 238,  258-259 
Job  Performance  Aids  (JPAs),  I: 

231,  233-255,  236,  242,  245-249 
Job  Performance  Measures  in,  I: 

232 

outputs,  I:  262-263 
pe'"Sonnel  data  factor  in,  I;  249- 
250 

procedures,  1:  242-263 
Resident  Schools  (RS),  I:  231,  238- 
239,  259 

revision  of  task  assignments,  I: 
260-262 

Self-Teaching  Exportable  Packages 
(steps),  I:  231,  235,  237,  250- 
255 

work  experience,  I:  239-240 
Instructional  Systems  Development  (ISO) 
program,  documentation  for,  I: 
83-84,  87 

initiation  of,  I:  1-5 
purpose  of,  I:  2,  214 
rationale  for  sequence  of,  I:  5-6 
Instructional  television  (ITV),  III: 
194-195 
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Job  Performance  Measures  (con- 
tinued) 

in  selecti ng  instructional 
settings,  1:  232,  242 
in  system  revision,  V;  110-111 
in  testing,  II:  44 
Job  Performance  Measures,  con- 
struction of,  I:  156-212 
checklists  for,  I:  196-197,  211 
conditions,  I:  177-178 
cues,  I:  179 

cut-off  scores  for,  I:  200-201 
data  collection  for,  I:  45,  46 
documentation  for,  I:  191-193, 
209 

errors  in  measurement,  I:  201- 
205 


outputs,  i:  209 
part-task  testing,  I:  185-187 
physical  fidelity,  I:  161-163, 
167 


predictive  validity,  I:  158-161, 
167 


procedures  for,  I;  165-209 
process  and  product  rating,  I: 
170-171,  195-196 
rationale  for,  I:  156-157 
rating  scales  for,  I:  197-200 
sample  list  of  validated  JPMs, 
I:  210 

sampling  plan,  I:  187-191 
scoring  procedures,  I:  191, 
194-204 

simulator  requirements  for,  I: 
163,  172-176 
standards,  I:  180-184 
testing  constraints,  I:  166- 
170 

tryout  procedure,  1:  207-208 
types  of  tasks  measured,  I: 
164-165 

validation  and  revision,  I: 
205-209 

Job  Performance  Test  (JPT),  I:  158 
Jury-of-Experts , I:  39-40,  41,  67 


Knowledge  of  results  (KOR),  III: 
31.  32,  33,  48,  51 


Learning  activities,  see  Learning 
events/activities,  specifying 


Learning  analysis,  II:  18-30 
attitude  category,  II:  28-30 
information  category.  It:  23-26 
mental  skills  category,  II;  19-23 
physical  skills  category,  II:  26- 
28 

Learning  categories  and  sub-categories, 
II:  16-17,  III:  11-17 
Learning  events/activities,  specifying, 
III:  1-105 

classification  of  learning 
objectives,  III:  10-17 
conditions  (active  vs.  passive), 

III:  4-5 

cues  or  prompts,  III:  6-8 
directions  for,  III:  24-26 
feedoack.  III:  6,  9-10 
flowcharts  (algorithms).  111:  18, 

20,  71-104 

guidelines  for  eleven  sub- 
categories of  learning.  III: 
17-18,  35-70 

learning  categories  with  sub- 
categories, III:  11-17 
learning  guidelines  (general),  III: 
3-6 

outputs.  III:  26 
procedures.  III:  3-26 
purposes,  III:  1-3 
learning  guidelines,  fut.ctions  of, 

III:  1-4,  208-213 
general.  III:  3-6,  10 
specific,  for  eleven  lefirning 
sub-categories.  III:  17-20, 

35-70 

sub-category  1:  rule-learning  and 
using.  III:  35-37,  71-73 
sub-category  2:  classifying.  III: 
38-40,  74-75 

sub-category  3:  identifying  symbols, 
III:  41-42,  76-77 
sub-category  4:  detecting.  III: 
43-45,  78-82 

sub-category  5:  making  decisions, 
III:  46-48.  33-86 

sub-category  6;  recalling  bodies  of 
knowledge,  III:  49-52,  87-90 
sub-category  7:  performing  gross 
motor  skills.  III:  53-55,  91-93 
sub-category  8:  steering  and  guid- 
ing, III:  56-57,  94-95 
sub-category  9:  positioning  move- 
ment, III:  58-62,  96-99 
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Learning  guidelines  (continued) 

Media  (continued) 

1 

sub-category  10:  voice  communi- 

decision matrices  for  selection 

1 

cating,  III:  63-65,  100-102 

of.  Ill:  172-184 

1 

sub-category  11:  attitude 

evaluation  of,  see  Existing 

1 

learning,  III;  66-70,  103- 

materials,  review  of 

1 

104 

revision,  III:  206-208 

; 

Learning  objectives  (LO),  II:  1, 

selection  (for  instructional 

4-5,  8-30,  see  also  Objectives, 

management  plan).  III:  106- 

common-factor,  II:  90-92 

124,  172-184 

testing,  II:  36,  38,  39,  42-45, 

Media-managed  instruction,  III:  194 
Media  specialist  (production  person- 

49-50, 64 

Learning  Objective  Analysis  Work- 

nel), III:  227 

sheet,  II:  7;  III:  27,  109,  224- 

Military  Occupational  Specialties 

225 

(MOS),  I:  3,  8 

Learning  objectives,  sequencing  and 

Misclassifications,  in  testing,  II: 

structuring,  see  Sequence  and 

52-55 

structure  of  learning  objectives. 

Mnemonics,  III:  31,  36,  41,  50,  59 

determining  | 

Learning  Resource  Centers  (LRCs),  1 

III;  141 

Non-equipment-oriented  jobs,  I:  79- 

Learning  steps  (LSs),  II:  4,  5,  36, 

81 

38 , 57  1 

1 

Management  decisions,  see  Decisions, 

Objectives,  development  of,  II:  1- 
34 

management 

action  statement,  II:  5-12 

Management  guidelines,  III:  124-137 

conditions  statement,  II:  5-6, 

Management  plan,  see  Instructional 

9-10,  13-14 

management  plan  and  delivery 

criteria  for  Terminal  Learning 

system 

Objective  and  Learning  Objec- 

Management plans,  student.  III:  185- 

tive  statements,  II:  8-16 

196 

inputs,  II:  1-2 

combination  plans.  III:  196 

Job  Performance  Measures  in,  II: 

computer-managed  instruction. 

2 

III:  19i-194 

learning  analysis  for  each  Term- 

contingency-managed i nstruction. 

inal  Learning  Objective,  II: 

III;  187-189 

18-30 

instructor-managed  instruction. 

learning  analysis:  attitude 

III:  186-187 

category,  II:  28-30 

media-managed  instruction.  III: 

learning  analysis:  information 

194 

category,  II:  23-26 

peer-managed  instruction,  III: 

learning  analysis:  mental  skills 

190-191 

category,  II:  lS-23 

student  self-managed  instruction, 

lea’-ning  analysis:  physical 

III:  195 

skills  category,  II:  26-28 

Marginal  students,  guidelines  for 

learning  categories,  II:  16-17 

managing,  III:  129-130 

learning  objectives  (LOs),  II: 

Materials,  existing— review  of. 

4-5,  8-30 

see  Existing  materials,  review  of 

learning  steps  (LSs),  II:  4-5 

Matrices  (for  media  selection).  III: 

outputs,  II:  30 

113,  115-118,  172-184 

procedures,  II:  6-30 

Media,  comparative  costs.  III:  112 

standards  statement,  II:  5-6, 

9-10,  14-16 
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Objectives,  development  of  (con- 
tinued) 

terminal  learning  objectives 
(TLOs)  preparation,  II:  2-5, 
6-16 


Peer-managed  instruction.  III:  190- 
191 , 215 

Perceptual  set(s).  III:  46,  63 
Performance  Evaluation  Plan,  V:  22- 
28 

area  of  "entry  skills,"  V:  24- 
25 

area  of  "external  requirements," 
V:  23-24 

area  of  "tests,"  V:  25-26 
area  of  "time  required  to  com- 
plete instructional  units," 

V:  27-28 

Performing  gross  motor  skills: 
learning  sub-category  7,  III: 
53-55,  91-93 

Photographer  (production  personnel), 
III:  228 

Physical  fidelity,  of  Job  Perform- 
ance Measures,  I:  161-163,  167 
Platform  lectures,  III:  248 
Positioning  movement  and  recalling 
procedures:  learning  sub-category 
9,  III:  58-62,  96-99 
Post-feedback  delay.  III:  61 
Posttest,  V:  109-110 
Predictive  validity  of  Job  Per- 
formance Measures  , I:  158-161, 

167 

Predifferentiation  of  stimuli.  III: 
38,  41,  49 
Pretest,  II:  73-75 
Print  specialist  (production  per- 
sonnel), III:  228 
Process  Evaluation  Plan,  V:  14-22 
Process  rating,  I:  196 
Product  rating,  I:  195 
Program  Evaluation  Review  Tech- 
nique (PERT),  V:  10 
Programmed  instruction,  III:  246- 
247 

Progress  Evaluation  Plan,  V:  11-14 


Questionnaire  survey,  administra- 
tion of,  I:  72-73,  106-112 


Questionnaire  survey  (continued) 
closed  form,  I:  38-39 
for  external  evaluation  data 
collection,  V:  73-74 
for  job  analysis,  I:  37-39,  44- 
46 

for  rating  tasks  for  training, 
1:  133-136 
open  form,  I:  38-39 
preparation  of,  I:  63-71,  134- 
136 

sample  selection  for,  I:  71-72, 
133-134 


Random  grouping.  III:  125 
Rank-order  testing,  II:  52-55 
Ratings  (Navy),  I:  3,  8 
Rating  scales  (for  Job  Performance 
Measures),  description  of,  I: 
197-198 

graphic,  I:  198-200 
numerical,  I:  197 
Recalling  bodies  of  knowledge: 
learning  sub-category  6,  111: 
49-52,  87-90 

Reinforcement;  reinforcer,  III: 
187-189 

Resident  Schools  (RS),  I;  238- 
239,  259;  III:  139-141 
Response  biases.  III:  46 
Revision  of  system,  see  System 
revision 

Revision  plan  format,  V:  113 
Rule-learning  and  using:  learning 
sub-category  1,  III:  35-37,  71-73 


Sampling  plan  for  Job  Performance 
Measures,  I:  187-191 
Self-pacing  instructional  plan,  III: 
124,  125-126 

Self-Teaching  Exportable  Packages 
(STEPS),  I:  231,  235-237,  250- 
255;  III:  140  12,  248-249 

Sequence  and  structure  of  learning 
objectives,  determing,  II:  79-96 
conmon-factor  learning  objectives, 
II:  90-92 

determining  relationships,  II: 
81-82,  83 

grouping,  II;  92-94 
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Sequence  and  structure  of  learning 
objectives,  determining  (contin- 
ued) 

outputs,  II:  94 
procedures,  II:  81-93 
purposes  of,  II:  79-80,  88-89 
with  dependent  relationships, 

II:  82-87 

with  independent  relationships, 
II:  90 

with  supportive  relationships, 
II:  87-88 

Setting,  see  Instructional  set- 
ting, seTecti on  of 
Shaping,  III:  55,  57 
Simulator  requirements  in  Job 
Performance  Measures,  I:  163, 
172-176 

Slide- tape  production.  III:  239- 
244 

Soft  data,  V:  4 

Source  materials,  of  job  analysis 
data  and  job  information,  I: 
97-98 

of  training  courses  and  in- 
structional materials,  I: 
95-97 

Staff  training,  instructional, 

IV:  7-12 

Standards,  Job  Performance 
Measures,  I:  180-184 
task,  I;  28-33 
terminal  learning  objective 
test,  II:  47-50 
training,  II:  47-50 
State-of-the-art,  III:  120 
Steering  and  guiding:  learning 
sub-category  8,  III;  56-57, 

94-95 

Stimulus  criteria  (media).  III: 
107-108 

Storyboards,  III:  234,  239-242, 
245-246 

Student  management  plans,  see 
Management  plans,  student  and 
Instructional  managen.ent  plan 
Student  self-managed  instruction, 
III:  195,  215 
Students'  Guide,  III:  269 
Students'  manual,  IV:  6 
Subject  Matter  Experts  (SMEs), 

I:  79,  81-82,  133 


Supplar.entary  Instruction,  III; 

250 

Syston  Master  Plan  (SMP),  III;  105, 
107,  138-142;  V:  10,  90-91 
System  revision,  V:  87-120 

based  on  changes  in  doctrine  or 
content  of  DOS,  V:  94 
based  on  efficiency  of  instruc- 
tion, V:  96-103 
based  on  External  Evaluation 
Report,  V:  95-96 
based  on  Internal  Evaluation 
Report,  V:  05 
description  of,  V;  87-89 
follow-up  activity,  V:  115 
for  improvement  of  instructional 
effectiveness,  V:  109-112 
for  operating  instruction,  V: 
105-106 

for  time  reduction,  V:  106-109 
guidelines  for  determining  revi- 
sion needs,  V:  103-112 
outputs,  V:  115-116 
preparation  of  revision  plan, 

V:  112-114 

priority  ratings  for,  V:  92 
procedures  for,  V:  93-115 
purposes  of,  V:  37-90,  93 
sources  for,  V:  90-91 


Task(s),  I:  12 

checklist  task  inventory,  I: 
152-154 

clustering  of,  I:  244-245 
conditions  statement  for,  I:  24- 
26,  89-93 

delay  tolerance,  I:  123-125,  246 
guidelines  for  diagramming,  I: 

24 

inventory,  verifying  and  validat- 
ing, I:  66-68 

Job  Performance  Measures  for,  I: 
157-211 

multiple,  I:  164-165 
statements,  I:  13-17,  61 
unitary,  I:  164 

validated  task  list,  sample,  I: 
33,  86 

Task  selection,  I:  113-155 

criteria  for,  I:  118-133,  152- 
154 
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Task  selection  (continued) 
data  assessing  and  decision- 
making  for,  I:  139-148 
data  collection  for,  I;  134-136 
data  consolidation  for,  I:  136- 
139 

management  constraints,  I:  143- 
144 

outputs,  I:  117,  148-149 
orocedures,  I:  117-148 
rationale  for,  I:  I lb- 11/ 
survey  sources,  I:  133-134 
Television  producer  (production 
personnel).  III:  229 
Television  program  production,  HI- 
244 

Terminal  learning  objectives  (TLOs). 
II:  1-31 

action  statement  of,  II:  5-12 
conditions  statement  of,  II:  5-6. 
9-10,  13-14 

guidelines  for  learner,  III:  4 
learning  analysis  for,  II:  18-30 
learning  categories,  II:  16-17 
standards  statement  of,  II:  5-6, 
9-10,  14-16 

Test  development,  II:  35-61 
attitudes  testing,  II:  45-46 
information  testing,  II:  42-44 
inputs,  II:  7 

mental  skills  testing,  II:  39-42 
misclassifirations,  II:  50-52 
outputs,  II : 37,  60 
physical  skills  testing,  II:  44-45 
procedures,  II:  38-60 
purposes,  Ii;  35 
rank-order  testing,  II:  52-55 
scoring,  II:  55-56 
standards,  II:  47-50 
types  of  tests,  II:  35,  38-39 
Testing  constraints  in  Job  Perform- 
ance Measures  , I:  166-177 
Trainer  Appraisal  Kit  (TAK),  IV: 

7-12 

Trainer  Development  Program  (TRADEP) 
IV:  9 

Training  Extension  Course  (TEC), 

Army,  III:  141-142 
Training  programs,  selection  of 
tasks  for,  1 : 114-133,  139-148 
Training  task  categories,  see 
Learning  categories 


Trial  report  (small  group),  example. 
Ill:  340-345 


Voice  communicating:  learning  sub- 
category 10.  Ill:  63-65,  100- 
102 


Writer  (production  personnel),  tu; 
229 


